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Abstract 

^  f%/s  -La 

We-  considers* algorithms  for  flow  control  in  computer 
networks  with  fixed  routing.  The  goal  is  to  establish  input 
rates,  for  each  source-destination  pair,i>tthat  satisfy  a 
particular  fairness  criterion/^  We  describe^several  algorithms 
in  which  the  input  rates  are  calculated  based  on  controls 
established  by  the  links  of  the  network.  These  controls  are 
updated  ^iteratively ,  using  feedback  information  from  the 
network.  “-We1  showrAthat  the  rates  thus  calculated  converge  to 
the  desired  values  when  the  links  are  assumed  to  update 
synchronously,  and  without  feedback  delay.  A  model  for 
asynchronous  operation  with  delay  is  given,  and  we  demonstrate 
for  this  model  that  the  input  rates  calculated  by  the 
synchronous  algorithms  may  fail  to  converge.  We  show  how  to 
modify  the  algorithms,  by  the  introduction  of  an  update 
protocol  and  by  using  more  of  the  available  feedback 
information,  so  that  convergence  of  the  rates  is  guaranteed. 

t  /  '  <*  .•  " 

-We  extend^  the  model__for  asynchronous  computation 
developed  by  Bertsekas '"’[14 ]xto  get  some  results  relating  to 
general  asynchronous  distributed  algorithms  with  update 
protocols.  These  results  are  used  to  give  an  alternate  proof 
of  the  correct  operation  of  one  of  the  flow  control 
algorithms. 

We  devele^  a  computer  program*  to  simulate  the  flow 
control  algorithms  for  a  voice  packet  network.  The  simulation 
results  indicate  that  the  algorithms  behave  as  expected  for  a 
network  with  static  loads.  However,  when  input  loads  change 
in  imitation  of  real  conversations,  the  control  algorithms  do 
not  adapt  fast  enough  to  control  the  flows  effectively.  s _ 
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Chapter  1 


Introduction 

1 . 1  Background 

Advances  in  packet  switching  techniques  make  packet 
switching  a  cost  effective  method  for  handling  sporadic  or 
bursty  comminications  traffic.  The  sporadic  nature  of  voice 
and  the  desire  to  integrate  voice  and  data  in  computer 
communication  networks  [1]f  [2],  makes  the  idea  of  packet 
voice  attractive.  In  this  thesis,  we  consider  the  problem  of 
limiting  traffic  flow  in  such  integrated  networks. 

In  a  traditional  circuit  switched  voice  communication 
network,  a  given  2-way  conversation  is  allotted  two  dedicated 
channels.  But  usually  a  user  spends  50$  of  his  time 
listening.  In  addition,  pauses  between  words  and  phrases  in 
the  speech  of  the  active  user  represent  a  source  of  wasted 
channel  resources.  These  smaller  units  of  uninterrupted  voice 
are  called  "talk-spurts".  The  random  nature  of  talk-spurts 
has  long  been  exploited  by  the  Bell  System  in  their  TASI 
algorithm,  used  on  intercontinental  lines  C  3  3 •  Digital 
variations  on  TASI  include  Digital  Speech  Interpolation  and 
Speech  Predictive  Encoding  CM],  [5]. 


In  an  integrated  voice  and  data  communication  network,  it 
is  necessary  to  adopt  flow  control  measures  to  limit  the 
amount  of  information  entering  the  network,  and  prevent 
congestion.  While  flow  control  techniques  for  data-only 
networks  have  reached  a  high  level  of  sophistication  [6], 
little  is  known  about  flow  control  for  voice.  The  different 
delivery  requirements  of  voice  and  data  demand  a  different 
approach  to  the  problem  of  flow  control  for  each.  While 
considerable  delay  may  be  acceptable  in  a  data  packet,  the 
same  delay  would  cause  a  voice  packet  to  be  discarded  by  the 
receiver  as  "too  late".  Conversely,  voice  may  suffer 
considerable  degradation  due  to  errors  and  still  be 
intelligible,  while  the  same  errors  in  a  data  packet  make  it 
worthless. 

Traditional  methods  of  flow  control  for  voice  simply 
block  the  initiation  of  new  calls.  TASI  type  systems  may  even 
block  new  talk-spurts,  resulting  in  clipping  of  the  received 
signal.  Loss  of  more  than  about  .51  of  the  signal  by  clipping 
has  proved  unacceptable. 

The  idea  of  embedded  coding,  first  proposed  at  the  Naval 
Research  Laboratory  [7],  provides  a  new  approach  to  voice  flow 
control  [8].  In  embedded  coding,  speech  is  encoded  into 
priority  ranked  packets.  The  lower  priority  packets  can  be 
discarded  as  needed,  while  the  remaining  packets  still  provide 
an  intelligible,  though  degraded,  signal.  Hence,  we  have 


traded  clipping  for  distortion.  The  level  of  network 
congestion  that  results  in  unacceptable  clipping  for  a  call 
blocking  scheme  is  much  lower  than  that  required  to  render 
embedded  coding  speech  unintelligible. 

Low  priority  packets  can  be  discarded  at  their  point  of 
entry  into  the  network,  as  well  as  at  the  point  of  congestion, 
resulting  in  a  variable  rate  encoding  scheme.  Clearly,  it  is 
better  to  discard  entering  packets  when  possible,  to  prevent 
unnecessary  waste  of  network  resources. 

An  algorithm  for  voice  flow  control  using  embedded  coding 
has  been  studied  by  a  group  at  Lincoln  Laboratories,  using  a 
computer  simulation  [8].  In  their  model,  conversations  are 
conducted  over  fixed  routes.  Low  priority  packets  are 
discarded  at  congested  nodes,  and  terminals  report  the 
received  rates  to  senders,  which  reduce  their  input  rates 
accordingly.  Provisions  are  included  for  allowing  the  senders 
to  increase  their  rates  when  the  network  is  lightly  loaded. 

The  primary  objectives  of  this  scheme  are  to  maintain 
stable  operation  of  the  network,  while  preventing  excessive 
delays  due  to  congestion  and  providing  the  highest  level  of 
service  possible  to  each  user. 

This  last  criterion  gives  rise  to  the  question  of  how  to 
allocate  network  resources  in  a  "fair"  manner,  while  giving 


everyone  the  best  possible  service.  Hayden  [9]  and  Jaffe  [10] 
simultaneously  and  independently  arrived  at  a  concept  of 
"fair"  rate  allocation,  which  was  generalized  by  Gafni  and 
Bertsekas  [11],  [12],  and  is  defined  in  the  next  section. 

1 .2  Problem  Model 

We  will  use  the  following  model  to  study  the  problem  of 
voice  flow  control.  Consider  a  network  which  consists  of  a 
set  of  links  and  sessions  ^  ,  where  a  session  is  a 
source-destination  pair  between  which  a  conversation  is  taking 
place.  Each  link  j  has  an  associated  capacity  Cj (  where  Cj20. 
Each  session  is  assigned  a  path  through  the  network,  which  is 
fixed  for  the  duration  of  the  conversation.  We  denote  by  ^ 
the  set  of  links  in  the  path  of  session  i.  We  denote  by^^ 
the  set  of  sessions  whose  path  contains  link  j. 

Let  ri(t)  be  the  input  rate  of  session  i  at  time  t.  For 
now,  we  assume  instantaneous  propogation  of  data  through  the 
network,  so  that  the  component  of  flow  on  link  j  at  time  t, 
due  to  session  i,  is  ri(t).  (This  assumption  will  be  dropped 
later,  in  favor  of  a  more  realistic  model.)  Then  we  define 
the  flow  on  link  j  at  time  t  as 

fj(t).  2.  r,(t).  (1.1) 

We  control  the  flow  only  by  limiting  inputs  rates,  and  not  by 
discarding  packets  when  the  links  are  congested.  We  wish  to 


control  the  input  rate  for  each  session  i  and  the  flow  on  each 


link  j  so  that  the  steady  state  rate  ri=  nm  ri(t)  and  steady 
state  flow  fj_  fj(t)  exist,  and  satisfy  the  constraints 
for  a  fair  allocation,  as  outlined  below. 

We  would  like  for  a  fair  rate  allocation  to  be 
indifferent  to  the  geographical  separation  of  the 
source-destination  pair.  While  priorities  may  be  established 
for  certain  sessions,  it  should  not  be  on  the  basis  of 
distance.  Furthermore,  two  sessions  of  the  same  priority 
should  be  assigned  the  same  rate,  if  the  rate  of  one  can  be 
traded  for  the  other,  without  reducing  the  rate  of  any  other 
session  or  violating  other  system  constraints.  This  will  make 
the  network  transparent  to  the  session,  in  the  sense  that  he 
cannot  tell  the  length  of  the  assigned  path  by  the  assigned 
rate. 

We  also  require  that  each  user  be  assigned  the  highest 
possible  rate,  while  guaranteeing  that  the  steady  state  flow 
on  each  link  does  not  exceed  a  given  function  of  the  link 
capacity. 

With  this  motivation,  we  give  the  definition  of  a  fair 
allocation,  first  presented  by  Gafni  [11].  First  we  need  the 
following  two  definitions. 


A  vector  xs(x.  ...,xn)  is  said  to  be  lexicographically 


less  than  or  equal  to  y= , . . . ,yn)  if  xi>y*  implies  the 


existence  of  j<i  such  that  xJ<yj.  We  write  this  asxiy. 

Given  a  vector  x£  Rnf  let  /x'>  denote  a  vector  whose 
coordinates  are  some  permutation  of  the  coordinates  of  x.  If 
the  coordinates  of  ^  have  the  property  that  xii,X2i. .  .ixn,  we 
call  x  the  Increasing  permutation  of  x,  and  we  denote  this 
vector  by  T.  (Note  that  Ti  for  any  permutation  'x  of  x.) 

Definition.  Let  X  be  any  subset  of  Rn.  We  say  that  x  e- X  is  a 

fair  allocation  over  X,  if  for  all  yeX,  "y*  i  x7 

Mjc 

We  may  think  of  X  as  a  "feasible"  set.  The  fair 
allocation  vector  solves  the  following  set  of  nested  problems. 
The  first  problem  is  to  find  a  subset  X1  0f  x,  such  that  the 
minimum  coordinate  of  a  vector  x-€X1  i5  greater  than  or  equal 
to  the  minimum  coordinate  of  any  vector  in  X.  Next  we  find  a 
subset  X2  0f  Xj ,  such  that  the  second  smallest  coordinate  of 
x£X2  is  maximized  over  Xj,  and  so  on. 

1 • 3  Pr&YiQua  Work 

Hayden  [9]  gives  a  distributed  algorithm  which  produces  a 
rate  vector  r=( . . . , r^f . . . )  that  is  a  fair  allocation  over  the 
set  defined  by 

fj<  ajcj  Vj€X,  (1.2) 

where  a  is  some  constant,  0<a^1.  The  rationale  behind  (1.2) 


is  simple:  we  restrict  the  steady  state  flow  on  each  link  to 
some  fixed  fraction  of  link  capacity,  reserving  the  unused 
capacity  as  a  buffer  against  transient  fluctuations  in  flow. 
We  call  a^cj  the  effective  capacity,  and  henceforth,  when  we 
refer  to  a  link's  capacity  it  is  assumed  that  we  mean  the 
effective  capacity. 

Jaffe  [10]  gives  an  algorithm  such  that  the  vector 
( . . . ,  bin, . . . )  is  a  fair  allocation  over  the  set  defined  by 

VliCj-i-j  V  (1.3) 
where  iS  SOme  positive  constant  associated  with  session  i. 
Hayden  offers  a  distributed  algorithm  for  achieving  Jaffe' s 
desired  rate  vector. 

The  rationale  behind  (1.3)  is  more  subtle  than  for  (1.2). 
First,  it  allows  us  to  establish  different  priorities  among 
sessions,  as  characterized  by  the  constant  bi#  Second,  it 
provides  a  buffer  against  transient  flows  which  is  sufficient 
to  allow  session  i  on  link  j  to  increase  its  rate  by  a  factor 

of  (1  +  bj.),  while  still  guaranteeing  that  fj<Cj. 
Alternatively,  it  permits  a  new  session  to  be  added  to  the 
link,  provided  its  rate  is  no  greater  than  that  of  the  roost 
privileged  session  already  using  the  link. 

Gafni  [11]  further  generalizes  the  feasible  set 
considered  by  Jaffe.  For  each  link  j  and  session  he 
introduces  functions  gjjR+«->R+  and  bi;R+— ^R+.  His  objective 


is  to  generate  a  rate  vector  r  such  that  the  vector 


( •  •  • » bi(r  j,)  f . . . )  is  a  fair  allocation  over  the  set  defined  by 


(1.4.1 ) 

and 

ri±0 

(1.4.2) 

and 

fj*cj 

Vjejf. 

(1.4.3) 

We  will  refer 

to  the  functions  gj(*)  and  b^(*)  as 

the  link 

constraint 

functions  and  session  constraint 

functions, 

respectively. 

For  convenience,  we  will  denote 

the  vector 

( •  •  • » b^(r , . . . )  by  Jj(r). 


In  order  to  guarantee  the  existence  of  a  unique  fair 
allocation  vector  over  this  set,  the  following  assumption  is 
needed  [12]: 

Assumption  1.1:  For  all  gw*)  is  monotonically 
non-decreasing,  and  for  all  i£&,  bi(«)  is  continuous, 
monotonically  increasing  and  maps  R+  onto  R+. 

We  note  that  this  assumption  also  implies  the  existence  of 
b”1(*).  Unless  otherwise  noted,  when  discussing  functions 
gj(«)  and  b^(*),  we  assume  that  Assuption  1.1  holds. 


Gafni  gives  an  algorithm  that  produces  the  desired  rate 
vector,  provided  that  an  additional  assumption  about  the  link 
and  session  contraint  functions  is  satisfied. 
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Assumption  JLtl:  For  each  i£i,  j^it  the  function  h^C), 
defined  by  hjj(f)  =  bi"^ (gj(f) ) ,  is  convex  and  differentiable  on 
R+,  and  satisfies  hii(o)=0. 

By  allowing  the  function  b^(*)  to  be  non-linear,  we  gain 
flexibility  in  making  priority  assignments.  Gafni  also 
provides  some  examples  where  it  is  desirable  for  g.(*)  to  be 

J 

non-linear  as  well.  We  summarize  one  such  argument  below. 


Assume  that  the  bit  rate  for  each  session  i  is  a 

stochastic  process  with  mean  r±  and  standard  deviation  diri, 

where  d^o.  For  each  link  j,  define  Djs  max  d^.  For  a  given 

a 

link  j,  let  k  be  the  session  in^j  with  the  maximum  mean  rate 
•"j.  If  we  assume  that  fjicj,  then  by  the  independence  of  the 
rates  of  different  sessions,  the  standard  deviation  OT(f .)  Qf 

J 

the  flow  fj  satisfies 

O' 


iDj  ^ri 

2.  ri)rk 


— Dj>l c j rk  • 


If  we  choose 
bi(r)=r 


(1.5) 


(1.6.1 ) 


6.(f)sf2/(cHD  2) 


(1.6.2) 


\ 

I 


then  from  (1.4)  and  (1.5),  we  have 

f  j  j\CjnT 

— DjV  cj8j  (cj-f  j) 

=cj-fj.  (1.7) 

Hence,  proper  selection  of  the  function  g,(.)  can  guarantee 

sufficient  reserve  capacity  on  each  link  to  accomodate 
fluctuations  in  flow  at  least  as  great  as  the  standard 
deviation  of  the  flow. 


We  note  that,  while  the  class  of  feasible  sets  considered 
by  Gafni  is  more  general  than  the  class  of  feasible  sets 
considered  by  Hayden,  the  two  classes  are  disjoint,  because  of 
Assumption  1.2. 


In  addition  to  seeking  fair  allocations  over  different 
feasible  sets,  Hayden  and  Gafni  also  use  different  me^nods  of 
controlling  the  session  input  rates.  Both  algorithms  are 
designed  for  synchronous  operation,  such  that  at  each  unit 
interval  of  time  n,  each  session  calculates  a  new  input  rate 

^(n).  For  both  algorithms,  the  rate  vectors  r(n)  can  be 
shown  to  converge  as  n  goes  to  Infinity,  and  the  limit  vector 
r  is  a  fair  allocation  over  the  specified  set. 


In  Hayden's  algorithm,  each  link  j  calculates  a  control 
value  Pj(n+1)  at  time  n+1 ,  according  to  the  equation 

Pj(n+1 )=Pj(n)+(ajCj-f j(n) )/Wj,  (1.8.1) 


-  18  - 


’ *V*\  •"«  f 

.».•#»  ■  w •  w '  i. » 


where  Wj  iS  the  number  of  sessions  on  (or  "weight”  of)  link  j, 
and  Sj  iS  some  constant  satisfying  0<ajii.  Each  session  then 
adjusts  its  rate  so  that 

r<(n)=  min  p<(n).  (1.8.2) 

j  J 

Hayden's  algorithm  can  also  be  modified  to  give  Jaffe's 
desired  rate  vector  by  changing  the  control  update  equation  to 

Pj (n+1 )=Pj(n)+(ajCj-f j(n)-pj (n) )/(Wj+1 ).  (1.9) 

While  it  has  been  shown  that  the  rates  r^n)  f0r  Hayden's 
algorithm  converge  to  the  desired  fair  allocation,  the  control 
values  for  the  links  do  not  necessarily  converge.  If  there 
exists  a  link  such  that  all  its  sessions  are  controlled  by 
other  links,  then  the  flow  on  that  link  will  converge  to  some 
constant  less  than  the  capacity.  In  the  attempt  to  bring  the 
flow  up  to  capacity,  the  link  will  increase  its  control  at 

each  update  by  ^°j-f j(n))/Wj,  and  the  control  will  grow  to 
infinity.  This  can  create  serious  problems  when  a  new  session 
joins  the  network. 

Even  though  it  has  been  shown  that  Hayden's  algorithm 
converges  to  the  desired  rate  vector,  computer  simulations  of 
his  algorithm  exhibit  distressing  oscillations  in  the  link 
flows  when  inputs  are  changing.  We  suspect  this  is  caused  by 
the  failure  of  Hayden's  model  to  accurately  reflect  delays  in 
the  network:  the  delay  between  the  time  that  a  link  updates 
its  control  and  its  sessions  learn  the  new  value,  and  the 


delay  between  the  time  that  a  session  changes  its  rate  and  the 
flow  on  any  of  its  links  reflects  that  change. 

In  Gafni's  algorithm,  at  time  n,  each  link  j  calculates 
for  each  of  its  sessions  it  a  control  value  Pjj(n)  given  by 

Pi j(n)=aj(n) (hij(cj-f j(n) )-ri(n) )  (1.10.1 ) 

where 

aj(n)=  1/(1+21  hkj'(Cj-fj(n))  (1.10.2) 

and  Is  defined  as  in  Assumption  1.2,  and  h'jj(»)  is  the 

derivative  of  h^O.  Each  session  then  finds  its  new  rate 
according  to 

ri(n+1 )=ri(n)+  min  Pjj(n).  (1.10.3) 

If  we  define  bi(r)-r  and  gj(f)sf,  then  the  rate  vector  as 
given  by  (1.4)  is  the  same  as  Jaffe's,  and  (1.10.3)  becomes 

ri(n+1)sri(n)+  min  (cj-f j(n)-ri(n) )/(Wj+1 ) .  (1.11) 

Note  the  similarity  of  (1.11)  and  (1.9). 

The  essential  difference  between  these  two  techniques  is 

that,  in  Hayden's  algorithm,  memory  of  the  past  state  resides 
with  the  links,  while  in  Gafni's,  past  state  information  is 
stored  by  the  sessions.  In  Hayden's  algorithm  the  links 
calculate  their  new  control  values  in  terms  of  the  the  past 
control  values  and  the  past  flows  (rates),  while  the  sessions 
find  their  rates  in  terms  of  the  present  controls.  In  Gafni's 
algorithm,  the  sessions  calculate  their  new  rates  in  terms  of 


the  past  rates  and  the  past  controls,  while  the  links  find 
their  controls  in  terms  of  the  present  flows  (rates). 

It  is  because  new  rates  are  calculated  in  terms  of  old, 
that  Gafni's  algorithm  has  the  important  property  that  the 
flows  on  the  links  are  always  less  than  or  equal  to  the  link 
capacities,  provided  that  the  flows  were  less  than  or  equal  to 
the  capacities  initially.  Hayden's  algorithm  can  only 
guarantee  that  flows  are  less  than  or  equal  to  capacities  in 
the  steady  state. 

While  this  appears  to  be  a  serious  flaw  in  Hayden's 
algorithm,  it  may  provide  certain  advantages.  Assume  that  the 
network  in  question  is  an  integrated  voice  and  data  network, 
and  that  the  stated  capacity  of  a  link  c^  iS  not  the  link's 
true  capacity  but  the  portion  of  its  capacity  allocated  for 
carrying  voice  packets.  If  the  voice  and  data  packets  are 
queued  separately,  with  voice  being  given  priority,  the 
algorithm  could  produce  a  rate  assignment  which  would  generate 

flows  fj  in  excess  of  cj,  without  actually  causing  any  voice 
packets  to  be  queued.  At  the  next  iteration  of  the  control 
update,  the  control  for  such  a  link  would  be  greatly  reduced 
and  the  voice  flow  fj  would  drop  below  cj,  providing  the  extra 
capacity  needed  to  transmit  the  data  packets  that  were  queued 
at  the  last  step.  Hence,  the  time  average  rate  assignment  for 
a  given  session  is  likely  to  be  higher  for  Hayden's  algorithm 
than  for  Gafni's. 


w 


k 


Furthermore,  Gafni's  algorithm  has  not  been  shown  to 
converge  if  the  initial  rates  are  chosen  outside  the  feasible 
set.  Hence,  if  random  fluctuations  in  the  session  rates,  or 
the  initiation  of  a  new  session,  cause  flows  to  exceed 
capacity,  there  is  no  guarantee  that  the  rates  will  return  to 
values  inside  the  feasible  set.  In  the  example  above,  where 
g^O  is  given  by  (1.6.2),  when  the  flows  exceed  capacity, 
the  link  constraint  function  does  not  even  satisfy  Assumption 
1.1.  Because  Hayden's  algorithm  has  been  shown  to  converge 
from  any  initial  control  vector,  it  must  eventually  recover 
from  such  disturbances,  if  they  are  sufficiently  infrequent. 

Gafni's  algorithm  also  has  the  disadvantage  that  each 
link  must  know  the  function  b^*)  for  all  of  its  sessions. 
This  is  not  a  serious  drawback,  though,  since  in  practice 
there  will  probably  be  only  a  small  number  of  different 
priority  classes  in  use.  The  function  b^*)  will  be  the  same 
for  all  members  of  a  given  priority  class,  so  that  the  link 
only  needs  to  calculate  P^U)  for  each  priority  class. 

As  noted  previously,  both  Hayden's  and  Gafni's  algorithms 
are  designed  for  synchronous  operation,  with  all  sessions 
updating  their  rates  simultaneously,  making  actual 


Implementation  impractical.  However,  Gafni  and  Bertsekas  [12] 
were  able  to  show  that  Gafni's  algorithm  will  produce  a 
sequence  of  rate  vectors  that  converge  to  the  desired  fair 


rate  allocation,  even  under  certain  asynchronous  conditions. 
Specifically,  they  consider  an  algorithm  where  a  single 
session  rate  r^  jS  updated  according  to  Cl. 10),  and  the  flows 
are  then  updated  to  reflect  the  change  in  r^.  This  process  is 
repeated  indefinitely,  with  each  session  updating  in  a  fixed 
cyclic  order. 

1 .4  Qy.ery.lfiM 

In  Chapter  1  of  this  thesis,  we  introduce  the  problem  of 
flow  control  for  packetized  voice  and  introduce  the  idea  of  a 
fair  rate  allocation  over  a  given  feasible  set.  We  describe 
previous  work  by  Hayden  and  Gafni,  each  of  whom  developed 
distributed  flow  control  algorithms  for  achieving  fair  rate 
assignments.  We  identify  some  problems  associated  with  their 
algorithms. 

In  Chapter  2,  we  describe  a  method  of  categorizing  flow 
control  algorithms  like  Gafni's  and  Hayden's.  We  show  how 
their  two  approaches  can  be  merged  to  unite  some  of  the 
advantages  of  each.  In  particular,  we  propose  two  algorithms 
which  produce  a  fair  rate  vector  for  sets  in  Gafni's  class  of 
feasible  sets  (1.4),  but  without  the  need  for  the  links  to 
calculate  separate  controls  for  each  priority  class.  We 
analyze  one  of  these  algorithms  in  detail. 


In  Chapter  3,  we  consider  how  the  model  of  Chapter  1 


fails  to  account  for  network  delays,  and  describe  some 
resulting  difficulties.  We  then  give  an  extended  model  which 
not  only  considers  delay,  but  also  asynchronous  operation  of 
the  flow  control  algorithms.  We  introduce  the  idea  of  an 
update  protocol,  which  permits  a  link  to  update  its  control 
only  when  the  protocol  is  satisfied.  We  use  update  protocols 
to  construct  some  asynchronous  flow  control  algorithms:  one 
that  gives  a  fair  rate  vector  over  Hayden's  feasible  set  and 
two  that  give  fair  rate  vectors  over  sets  in  Gafni's  class  of 
feasible  sets.  For  two  of  these  algorithms,  we  prove  that  the 
generated  control  sequences  converge  to  produce  the 
appropriate  fair  rate  vectors,  given  the  assumptions  of  the 
asynchronous  model. 

In  Chapter  4,  we  build  on  the  work  of  Bertsekas,  et. 
al.,  [13],  [14],  who  have  developed  results  that  apply  to 
general  asynchronous  algorithms.  Bertsekas  considers  a  system 
in  which  N  processors  find  an  element  of  a  given  solution  set 
by  iteratively  computing  estimates  of  the  solution.  Each 
processor  receives  feedback  measurements  from  the  system,  and 
uses  these  measurements  to  update  its  current  estimate.  In 
Bertsekas'  model,  a  processor  may  update  its  estimate  at  any 
time,  asynchronously  with  respect  to  the  other  processors.  We 
extend  the  model  to  include  algorithms  where  updates  times  are 
restricted  by  update  protocols.  We  give  a  theorem  similar  to 
Bertsekas',  describing  a  class  of  such  algorithms  for  which 
the  estimate  sequences  converge.  We  use  this  result  to  give 


an  alternate  proof  of  the  correct  operation  of  one  of  the  flow 
control  algorithms  given  in  Chapter  3.  We  also  give  a  theorem 
that  shows  how  a  synchronous  algorithm,  taken  from  a  given 
class  of  algorithms,  can  be  implemented  asynchronously  by  the 
addition  of  an  appropriate  update  protocol. 

In  Chapter  5,  we  describe  a  computer  program  written  to 
simulate  the  flow  control  algorithms  of  Chapter  3.  The 
program  simulates  a  network  carrying  voice  traffic  only,  where 
each  source-destination  pair  represents  a  voice  conversation. 
At  any  given  time,  one  member  of  each  such  pair  is  talking, 
and  the  other  silent.  We  study  the  steady  state  behavior  of 
the  network  by  setting  the  average  talk-spurt  duration  to 
infinity.  For  the  static  network,  the  algorithms  behave  much 
as  predicted.  We  also  set  the  average  talk-spurt  duration  to 
a  value  representative  of  actual  speech,  to  study  how  the 
network  behaves  under  real-life  conditions.  The  results  of 
these  simulations  are  inconclusive,  and  indicate  that  our 
model  is  not  detailed  enough  to  let  us  accurately  predict  the 
behavior  of  our  algorithms  in  dynamic  operation. 

In  Chapter  6,  we  summarize  our  results  and  give 
suggestions  for  further  research. 


Chapter  2 


Generalized  Synchronous  FI ow  Control  A1 eorlthms 

2.1  Options  Ion  Algorithm  Design 

As  mentioned  in  section  1.3,  the  essential  difference 
between  Hayden’s  algorithm,  given  by  Cl. 8)  and  Gafni's  in 
(1.10),  is  that  in  Hayden's,  state  is  stored  by  the  links  and 
in  Gafni's,  state  is  stored  by  the  sessions.  In  addition  to 
choosing  where  state  memory  resides,  the  algorithm  designer 
must  also  consider  how  to  allocate  the  burden  of  calculation. 
If  the  feasible  set  over  which  a  fair  allocation  is  sought  is 
of  the  form  given  in  (1.4),  both  the  constraint  functions 
gj(*)  and  b^(‘)  must  appear  somewhere  in  the  update  equations 
for  the  link  controls  or  the  session  rates.  Responsibility 
for  calculations  involving  g.(*)  may  be  given  to  either  the 
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sessions  or  the  links,  and  the  same  applies  to  b^c). 

As  an  example  of  what  this  means,  consider  the  following 
algorithm.  Let 

PjCn+1 )spj(n)+aj(n)(Cj-f jCn)-Pj(n))  (2.1.1) 

and 


ri(n)=  min  (gj(Pj(n) ) )  Vi£^ 


(2.1.2) 


where  {a j ( n ) }  iS  some  "appropriately"  chosen  sequence, 
designed  to  ensure  the  algorithm's  convergence.  This 
algorithm  is  the  reverse  of  Gafni's:  where  his  algorithm 
assigns  memory  to  the  sessions  and  calculation  to  the  links, 
this  algorithm  assigns  memory  to  the  links  and  calculation  to 
the  sessions. 

Of  the  eight  possible  ways  to  assign  responsibilty  for 
memory  and  calculation,  two  are  clearly  undesirable.  An 
algorithm  where  calculations  involving  the  link  constraint 
functions  are  performed  by  the  sessions,  with  the  links  doing 
the  calculations  involving  the  session  constraint  functions, 
obviously  entails  excessive  overhead. 

Ideally,  we  would  like  for  a  given  link  j  to  need  to  know 
only  its  own  constraint  function  g.(*),  and  for  a  given 
session  i  to  need  to  know  only  its  own  constraint  function 
b^*).  Such  an  algorithm,  with  state  memory  assigned  to  the 
links,  is  given  by 

Pj(n+1)=Pj(n)+aj(n)(gj(cj-fj(n))-Pj(n))  VjQf.  (2.2.1) 

and 

'Vn).  min  l>i"1(Pj(n)>  ViejU  (2.3.2) 

where  the  sequence  {aj(n)}  is  chosen  according  to  criteria 
discussed  in  section  2.4.  We  call  this  the  generalized  link 
memory  algorithm. 


Another  such  algorithm,  with  state  memory  assigned  to  the 
sessions,  is  given  by 

Pj  (n)sgj(cj-f  j(n) )  (2.3.1) 

and 

ri(n+1 )sri(n)+min  aj.(n)  (bj.”1  (Pj(n)  )-ri(n) )  Vi£$-  (2.3.2) 

tie* 

where  (a^(n)}  i5  some  appropriately  chosen  sequence. 

For  both  these  algorithms,  it  is  easy  to  see  that  if  the 
rates  and  controls  converge,  they  converge  to  values  that 
satisfy  (1.4.1).  The  problem  remains  to  choose  the  sequences 

(ai(n)}  or  (aj(n)}  in  a  manner  that  guarantees  that  the  rates 
converge  to  a  unique  point,  and  further  to  show  that  that 
unique  point  is  fair  over  the  set  defined  by  (1.4) 

In  general,  it  is  not  always  possible  to  select  a 
sequence  that  guarantees  rate  convergence.  The  choice  of  such 
a  sequence  depends  on  the  link  and  the  session  constraint 
functions,  and  on  which  algorithm  is  being  used. 

In  section  2.4,  we  give  a  definition  for  aj(n)  and 
conditions  on  gj(*)  and  b^*),  such  that  the  rates  produced  by 
the  generalized  link  memory  algorithm  (2.2)  can  be  shown  to 
converge.  We  have  not  yet  investigated  this  problem  for  the 
algorithm  proposed  in  (2.3). 

Before  showing  how  to  select  the  sequences  (a-s(n)}  for 
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the  generalized  link  memory  algorithm,  we  pause  to  discuss  the 
nature  of  the  limit  points  of  the  rate  and  control  sequences, 
and  to  prove  some  theorems  about  the  limit  points. 

2.2  The  fixed  Points  of  the  Rate  and  Control  Update  Equations 

Before  trying  to  prove  that  the  control  and  rate 
sequences  given  by  generalized  link  memory  algorithm  converge, 
we  consider  whether  or  not  there  exist  control  and  rate 
vectors  that  are  "fixed  points"  of  (2.2),  that  is,  we  want  £* 
and  x*  such  that  if  £(n)=£*  and  x(n)=x*,  then  £(n+1)=£*  and 
x(n+1 )=£*.  In  this  section,  we  give  a  centralized  algorithm 
for  finding  the  fixed  points  £*  and  x*  of  (2.2),  and  show  that 
they  are  unique. 

We  will  also  see  that  the  fixed  rate  vector  x*  is  the 
same  for  the  algorithms  given  by  (1.10),  (2.1),  (2.2)  and 
(2.3).  Since  each  of  these  algorithms  was  proposed  to  find 
the  fair  rate  allocations  over  the  same  set,  it  is  not 
surprising  that  they  have  the  same  x*. 

In  the  next  section,  we  show  that  jj*  and  £(x*)  are  fair 
allocations  over  the  appropriate  sets  for  the  generalized  link 
memory  algorithm. 

If  there  exist  vectors  and  x*  that  are  fixed  points  of 
(2.2),  then 
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(2.4.1) 


and 


ri»*  min  bi-’lPj*) 

(2.4.2) 

where 

f  j*s  21  r i*» 

(2.4.3) 

Combining  these  three  equations,  we  get 

ri«s  min  bi"1(8j(Cj- rk*>> 

(2.5) 

It  is  easy  to  see  that  similar  manipulations  of 

(1.10),  (2.1) 

and  (2.3),  all  yield  equation  (2.5). 

We  now  show  how  to  uniquely  construct  £*.  For  each  link 
j  define  Xj  such  that 

Xj*8j(cj-  21  bi”1(Xj))-  (.2.6) 

Assumption  1.1  guarantees  that  (2.6)  has  a  unique  solution. 
See  Figure  2.1.  Let  p'U  min  X.  iet  ©d.1  be  the  set  of  links  j 
for  which  Xj=p1,  and  let  be  the  set  of  all  sessions  on 
links  in  . 

Suppose  p*  exists.  By  (2.4)  we  have 

Pj«=gj(cj-21  bi"1^  “in  Pk»)).  (2.7) 

Therefore,  by  Assumption  1.1, 


(2.8) 


and  so,  from  Figure  2.1,  we  see  that  Pjt^Xj,  for  all 


Bounding  Pk«  by  xk  in  (2.7)  ,  we  have, 


Pj*i8j(Oj-21  bt-’t  *„)) 


for  and  so 


Pj*igj(cj-2.  bi-1(Xj)) 


sXJ 


(2.9) 


Thus  if  p«  exists,  p^e.  min  x^p1  for  all 

A. 


Now  we  may  rewrite  (2.7)  as 

pj»sgj(0j-2L  bi”1  (P1  bj-U  min  Pk«)).  (2.10) 

Equation  (2.10)  suggests  the  following  procedure.  To  find  the 
next  smallest  Pj«f  construct  the  reduced  network 

"7fs(X,'X,)»  where  and  the 

capacities  of  the  links  are  defined  by 

ci'=Cj-  22.  bj^"1  (p1 ) •  (2.11) 

Now  find  X.'  for  each  link  ,  and  plf=  min  X^'.  Repeat  this 
procedure  until  all  the  coordinates  of  £*  have  been  found. 
Thus,  by  construction,  p*  exists  and  is  unique. 


2.3 


Ida  fata  and  Caairpl  fixed  faints  am  fair 


In  this  section,  we  show  that  jj*  and  £*,  as  found  in 
section  2.2,  are  fair  allocations  over  the  sets  specified 
below.  This  and  the  results  of  the  last  section  imply  the 


existence  and  uniqueness  of  a  fair  allocation  rate  over  the 
set  given  by  (1.4).  This  last  result  was  also  shown  in  [12], 


Theorem  2.1  If  £*  and  £*  are  the  unique  fixed  points  of. 
(2.2),  then  £*  is  a  fair  allocation  over  the  set  defined  by 

Pj*6i(cj-  X.  “in  bi-1(pk))  (2.12.1) 

and  £(£*)  is  fair  over  the  set  defined  by 

bi(ri)<gj(cj-f  j)  VieJ,Vj^/i.  (2.12.2) 

Proof .  As  described  in  chapter  1,  the  fair  allocation  vector 
over  a  set  X  solves  a  nested  hierarchy  of  problems.  The  first 
problem  is  to  maximize  the  minimum  coordinate  of  vectors  in  X. 
Next,  we  maximize  the  second  minimum  coordinate  over  all 
vectors  which  solve  the  first  problem,  and  so  on.  Our 
algorithm  for  finding  £*  and  £*  solves  for  these  vectors  by 
just  such  a  nested  procedure,  finding  the  minimum  coordinate 
of  each  vector,  then  finding  the  next  smallest  coordinates, 
and  so  on.  Hence,  it  is  sufficient  to  show  that  the  first 
iteration  of  the  algorithm  maximizes  the  minimum  coordinates 
of  J2*  and  £(£*).  The  "correctness"  of  the  subsequent 
iterations  follows  by  induction. 

We  claim  that  p\  the  minimum  coordinate  of  £*  is  the 
maximum  minimal  coordinate  of  any  vector  £  in  the  set  given  by 
(2.12.1).  Suppose  otherwise.  Then  there  must  exist  &  in  the 
feasible  set  with  minimum  coordinate  q^>p^.  If  that  were  so, 


we  would  have,  for  each  link  j, 


P1  <Q1 


<gj(0j-  2  bi-U  «in  qk)) 

igj(Cj-  2_  bi'^q1)) 

•**<$*• 

But  this  is  a  contradiction,  since,  for  each  , 

p1=gj<°j-  2-  bi-'<pD>. 


(2.13) 


(2.14) 


Now,  since  bj-1(»)  is  strictly  increasing,  (2.2.2) 

implies  b1(ri«)  -  min  Pi«.  Because  p1  is  the  minimum 

i  «  &  J 

coordinate  of  £*,  the  minimum  coordinate(s)  of  J2(r*)  must  be 
bi(ri#)sp1,  for  each  iejJJ.  We  claim  that  this  choice 
maximizes  the  minimum  coordinate  of  £(.£*),  since,  if  it  did 


not,  there  must  exist  £(&)  in  the  set  given  by  (2.11.2)  with 
minimum  coordinate  bm(sm)>p^»  But  if  that  were  so,  we  would 
have  for  each  session  i,  for  each  link 

p,<w 

<»i<»i) 


igjCcj- 


2.  sk) 

Z.  bk-1  (p1 ) ) * 


(2.15) 


which  is  a  contradiction.  This  completes  the  proof  of  Theorem 


2.1. 
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Notice  that  we  have  not  yet  shown  that  £(l*)  is  fair  over 
Gafni's  feasible  set  (1.4),  as  desired.  We  cannot  show  this 
without  making  further  restrictions  on  b^(»)  and  gj(*),  since 

r^e  might  be  negative  for  some  i,  and  £(£*)  might  not  even  be 
an  element  of  the  set  given  by  (1.4).  This  cannot  happen  when 
Assumption  1.2  holds.  We  give  the  following  corollary. 

Corollary  2.1 .  For  each  define 

hij(x)sbi“1 ^8j(x)).  If  hij(0)=0,  then  is  fair  over  the  set 
defined  by 

Pj^giUj-  2.  min  bi"1(pk>>  (2.16.1) 

bi‘1(Pj)20  V'iSj,  (2.16.2) 

and 

S."1”  bi-1(|>k)i«j  Vifei.Vjeift,  (2.16.3) 

and  J}(£*)  is  fair  over  the  set  defined  by  (1.4). 

In  order  to  prove  the  corollary,  we  invoke  the  following 
lemma.  The  lemma  is  stated  without  proof,  since  it  follows 
trivially  from  the  definition  of  fair  allocation. 

Lemma  2.1  If  a  vector  x  is  fair  over  a  set  X,  and  Y  is  a 
subset  of  X,  then  x  is  fair  over  Y,  if  and  only  if  x<*  Y. 

Proof  of  Corollary.  By  the  lemma,  we  need  only  show  that  j>* 
and  £*  are  elements  of  the  appropriate  sets. 


(2.17) 


Recalling  (2.5),  £»  is  the  solution  to 


ri*=  min  hi j(ci-f j*), 

j«£. 

and  hence 


fj«.  *Z-  “in  hlk(ck-fk*) 

<  2-  hijUrfJ*> 

a .«r  • 


Now  suppose  fj»>cj20.  Then 


fj*<  hij(O) 

=0, 


(2.18) 


(2.19) 


which  is  a  contradiction.  Hence  for  all  j.  Since 

fj«icj,  (2.17)  gives  us  ^**0.  That  j?*  satisfies  (2.16) 
follows  trivially  from  (2.4)  and  the  fact  that  x*  satisfies 
(1.4).  This  completes  our  proof. 


Note  that  the  condition  hij(o)sO  is  sufficient  to  show 
that  £*  is  in  the  set  defined  by  (1.4),  but  it  is  not 
necessary . 

These  theorems  allows  us  to  concentrate  on  finding  a  fair 
control  vector,  rather  than  a  fair  rate  vector. 

2.4  The  Flow  Control  Algorithm 

The  genralized  link  memory  algorithm  is  not  completely 
specified  until  we  describe  how  the  sequences  {a^(n)}  are 
chosen.  One  obvious  way  to  choose  the  sequences  is  to  let 
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aj(n)sAj(pj(n)),  where  A j ( *  >  is  some  real  valued  function. 

In  order  to  establish  some  conditions  that  A.(*)  should 

J 

satisfy  to  guarantee  the  convergence  of  the  controls,  we 
consider  a  network  consisting  of  a  single  link  j.  For  such  a 
network,  (2.2)  becomes 

p(n+1 )sp(n)+A(p(n) )(g(c-  b.-l (p(n) )  )-p(n) ) .  (2.20) 

For  convenience,  define 

G(x)=g(c-  b.-1 (x) )  (2.21) 

and 

H(x)=x+A(x)(G(x)-x).  (2.22) 

Then  (2.20)  becomes 

p(n+1 )=H(p(n) ) .  (2.23) 

It  is  well  know  that  a  sequence  defined  as  in  (2.23)  will 
converge  for  any  intial  £(0),  if 

} H *  ( x )  |  <  1  Vx  (2.24) 

and  there  exists  x*  such  that  H(x*)=x*. 

While  the  condition  in  (2.24)  guarantees  that  the 
controls  converge  for  a  single  link  network,  we  are  not  able 
to  show  that  (2.24)  guarantees  convergence  for  a  more  general 
network.  We  can,  however,  prove  convergence  for  a  multi-link 
network  for  which  similar,  but  more  restrictive,  conditions 
hold. 

Theorem  2.2  Let  9?  =  (^  £J^)  be  any  network.  Let  g^*)  and 

J 


b^O  satisfy  Assumption  1.1.  For  each  j^^,  Sfrj£j,  define 

Gj(x,c,S)  =  gj(c-  2-  bi"1^))  (2.25) 

and 

Hj(x,c,S)=x+Aj(x)(Gj(x,c, S)-x),  (2.26) 

where  A^(x)is  a  continuous  function  such  that  0<Aj(x)^1  for 
all  x.  Suppose,  for  each  j,  Sj(‘)  is  uniformly  continuous, 
and 

0i^Hj(x,c,S)<1,  (2.27) 

for  all  x,  for  any  SC^j  and  c>  Oic^Cj.  Then,  for  any  initial 
control  vector  J2(0),  the  controls  j}(n)  and  rates  r(n)  given  by 

(2.2),  with  aj(n)=Aj(pj(n)),  converge  to  fair  allocations  over 
the  sets  given  by  (2.11). 

The  proof  of  Theorem  2.2  is  deferred  to  Chapter  3,  where 
we  will  see  that  it  is  a  special  case  of  a  more  general 
theorem  relating  to  asynchronous  flow  control  algorithms. 

The  conditions  of  Theorem  2.2  are  somewhat  restrictive 
and  we  believe  that  the  controls  and  rates  will  converge  for 
algorithms  where  Aj(*),  gj(*)  and  r^*)  are  less  cot.strained. 
Specifically,  we  conjecture  that  condition  (2.2 7)  can  be 
replaced  by 

j^HJ(x,c,s)|<1,  (2.28) 

but  we  have  been  unable  to  prove  so. 
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Let  us  consider  the  restrictions  that  (2.27)  places  on 


Aj(«),  gj(*)  and  b^(«).  While  it  is  difficult  to  describe  the 
entire  class  of  functions  H^c,*)  such  that  Aj(«)  may  be 
chosen  to  satisfy  (2.27),  at  least  one  sub-class  is  easily 
identifiable. 


Consider  the  class  of  functions  Hj(*,*)  such  that  Aj(«) 
may  be  chosen  to  be  a  constant,  that  is,  Aj(x)sAj,  and  such 
that  (2.27)  is  satisfied.  We  assume  that  0<Aj<i.  Then  (2.2 7) 
becomes 


“1<Aj(^Gj(x,c,s)-1)<0.  (2.29) 

By  Assumption  1.1,  ^Gj(x,c,S)  is  negative  for  all  x,  c  and  S, 
and  hence  the  right  inequality  of  (2.29)  is  always  satisfied. 
Rearranging  the  left  inequality,  we  get 


1-1/A 

J 


& 


Gj ( x, c, S) . 


(2.30) 


Hence,  for  those  functions  Gj(XfC,S)  whose  partial  derivatives 
with  respect  to  x  are  bounded  below,  the  conditions  of  (2.27) 
can  always  be  met  for  small  enough  A. 

J 


While  (2.30)  may  seem  a  bit  restrictive,  we  make  the 
following  observation.  Suppose  that  we  are  given  two  sets  of 
functions  gj(*)  and  gj > ( • )  that  satisfy  Assumption  1.1,  and 
such  that  g  j  ( x )  s  g  j  '  ( x )  for  x£[0,cj].  Then  regardless  of  how 
the  functions  may  differ  outside  that  interval,  the  fair  rate 
allocation  over  the  set  defined  by  (1.4)  and  gj(*)  la  the  same 
as  the  fair  rate  allocation  defined  by  (1.4)  and  gH*(*). 


Hence,  we  may  "tailor"  the  functions  gj(*)  any  way  we  like 

outside  fO»Cj]t  in  order  to  satisfy  (2.30),  without  affecting 
the  point  to  which  the  algorithm  converges.  Even  so,  there 
may  be  some  functions  b^(»)  and  gj(*)  for  which  (2.30)  will 
not  hold. 

2.5  Ad  Example 

In  the  last  section  we  described  how  the  functions  g.(*) 
may  be  tailored  outside  the  interval  [0,0^]  to  give  a  function 
which  will  satisfy  the  constraints  of  Theorem  2.2.  In  this 
section,  we  give  an  example  of  this  technique. 

Suppose  we  are  interested  in  finding  a  fair  allocation 
over  the  set  given  by  (1.4),  using  functions  b^*)  and  gj(*) 
as  defined  by  (1.6).  Recall  that  such  a  fair  allocation  has 
the  property  that  the  excess  capacity  on  any  link  is 
sufficient  to  handle  a  fluctuation  in  flow  as  great  as  the 
standard  deviation  of  the  flow.  For  convenience  define 
kjS1/(cjDj2).  Then  gj(f)=kjf2*  This  choice  of  gj(’)  is 
clearly  unsatisfactory,  since  it  is  neither  monotonically 
non-decreasing,  nor  uniformly  continuous.  We  propose, 

instead,  that  gj(’)  he  defined  as 

gj(f)s  Co  when  f<0 

]kjf2  when  0<f<c j  (2.31) 

(,2kjCjf-kjCj2  when  f>Cj. 

Now, 
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.V 


(2.32) 


Hj(x,c,S)=x+Aj(gj(c-Wsx)-x), 
where  Wg  iS  the  number  of  sessions  in  S.  So, 
j ( x» c» S)  =  1  -Aj (WsGj  '(c-Wsx)  +  1). 

Since 


gj'(f)sj^0  when  f<0 

when  0<f<Cj 

C2kjCj  when  f>cj 

we  have,  for  all  x, 

0<6j'(«)i2kjCj. 

Combining  (2.33)  and  (2.35),  we  get 
1-Aj(l+2kjCjWs)i  Hj(x,c,S)<1. 


Hence,  we  choose 

AjSi/(l+2kjCjWs) 

and  the  conditions  of  Theorem  2.2  are  satisfied. 


(2.33) 

(2.34) 

(2.35) 

(2.36) 

(2.37) 


Chapter  3 


Asynchronous  Flow  Control  Algorithms 

As  mentioned  earlier,  the  system  model  given  in  Chapter  1 
fails  to  accurately  describe  the  operation  of  flow  control 
algorithms  by  ignoring  communication  delays.  In  this  chapter, 
we  describe  in  detail  how  the  model  fails,  and  then  give  an 
improved  model  which  allows  for  feedback  delays  and 
asynchronous  operation.  We  introduce  the  idea  of  an  update 
protocol  and  give  some  examples.  We  give  an  improved  flow 
control  algorithm,  together  with  a  proof  that  the  rates  and 
controls  it  produces  converge  to  the  fair  allocation  over 
Hayden's  feasible  set,  under  the  assumptions  of  the 
asynchronous  model.  Finally,  we  modify  the  algorithm  to 
produce  fair  rates  and  controls  over  the  more  general  feasible 
set  defined  by  C2.11). 

3.1  £& fed back  Delays 


In  section  1.2  we  made  the  assumption  that  data 
propagates  instantaneously  through  the  network,  allowing  us  to 
define  the  flow  on  a  link  at  time  t  as 


fjU), 


i-l(t)  Vj€^. 


t 

v  *-•  *  v  . *  •  v  v  v  .*  .• 
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(3.1) 


We  also  assumed  that  at  time  t,  each  session  i  knows  the 
current  value  of 
min  pi(t) 

or 


as  required  for  the  calculation  of  ri(t). 


In  practice,  however,  the  concepts  of  instantaneous  flow 
and  rate  are  not  well  defined.  The  presence  of  queues 
distorts  the  input  rates  of  the  sessions  as  seen  by  the  links, 
and  even  in  the  absence  of  queues,  propagation  delays  prevent 
the  links  from  knowing  the  sessions'  current  input  rates. 
Hence,  we  cannot  expect  these  relations  to  hold  at  all  times 
t.  However,  for  the  flow  control  algorithms  previously 
discussed,  where  rate  and  control  updates  take  place  at  tnf 
0=0,1,...,  we  require  only  that 

21  ri(tn)  VjC^,  (3.2) 

and  that  each  session  i  knows  at  time  tn>  the  current  value  of 

1 

or 

We  outline  below  a  method  for  guaranteeing  that  (3-2)  holds. 


The  other  condition  can  be  met  only  by  careful  synchronization 
of  the  links  and  sessions,  when  selecting  the  update  times  t 


In  order  to  guarantee  that  (3.2)  holds  for  all  tn  we 
assume  that  the  links  calculate  their  flows  by  summing  the 
rates  of  their  sessions,  where  the  rates  are  communicated  to 
the  links  by  the  sessions.  While  this  eliminates  the  problem 
of  rate  distortion,  there  will  still  be  delays.  Communicating 
the  rates  requires  transmitting  additional  data  between  links 
and  sessions,  but  guarantees  that  (3.2)  holds,  provided  that 
the  interval  between  updates  is  long  enough  to  allow  the 
necessary  exchange  of  information. 

An  alternative  to  this  would  be  for  each  link  to  observe 
the  amount  of  traffic  it  carried  over  some  recent  interval  of 
time,  and  use  that  as  its  flow.  Observing  the  flow  has  the 
advantages  of  simplicity  and  low  overhead,  but  makes  it 
impossible  to  enforce  the  condition  (3.2),  because  of  rate 
distortion  by  the  queues. 

Hayden's  [9]  simulation  of  a  network  using  his  flow 
control  algorithm  assumed  that  flows  were  observed.  We 
believe  that  this  contributed  to  the  oscillations  in  flow  that 
his  simulation  displayed.  Another  contributing  factor  is  the 
delay  between  the  time  that  a  link  updates  its  control,  and 
its  sessions  learn  the  new  control  value.  To  see  how  such 
oscillations  might  arise,  we  consider  the  following  examples. 

Suppose  we  have  a  network  consisting  of  a  single  link, 


with  capacity  c=1,  serving  a  single  session.  If  we  seek  a 
fair  allocation  over  the  set  given  in  (1.2)  with  asl,  the 
update  equations  for  Hayden's  algorithm  are 

p(n+1 )=p(n)+c-f(n)  (3.3.1) 

and 

r(n)sp(n).  (3.3.2) 

Suppose  that  the  delays  are  such  that  f(n)=p(n-1),  instead  of 
f(n)=p(n).  Then 

p(n+1 )=p(n)-p(n-1 )+1 .  (3.4) 

For  p(0)=0  and  p(1)=1,  the  subsequent  controls  are  given  in 

Table  3.1.  Obviously,  the  controls  cycle  forever,  and  do  not 
converge. 

For  the  same  network,  suppose  we  want  a  fair  allocation 

over  the  set  in  (1.4),  with  g(f)*f  and  b(r)*r.  Using  Gafni's 
algorithm,  the  update  equations  are 

p(n)=  .5(c-f (n)-r(n) )  (3.5.1) 

and 

r(n+1 )=r(n)+p(n) .  (3.5.2) 

If  the  flow  is  communicated,  f(n)=r(n).  However,  feedback 
delay  can  cause  the  session  to  learn  the  link  control  value 
late,  so  that 

r(n+1 )=r(n)+p(n-1 ) 

=r(n)+  . 5( 1-2r(n-1 ) ) 

=r(n)-r(n-1 )+  .5  .  (3.6) 


For  r(0)=0  and  r(1)s.5,  the  subsequent  rates  are  given  in 
Table  3.2.  Again  we  see  that  the  rates  cycle,  and  do  not 


Table  3.1 


Table  3.2 


Divergent  Example  for 
Hayden's  Algorithm 


Divergent  Example  for 
Gafni's  Algorithm 


converge. 


These  simple  examples  show  how  important  it  is  to  account 
for  feedback  delay  in  our  system  model. 

3.2  IhS.  Asynchronous  System  Model 

In  this  section,  we  give  a  model  for  studying  systems 
with  feedback  delay  which  use  asynchronous  flow  control 
algorithms.  The  model  assumes  that  flows  are  communicated  and 
that  state  memory  is  assigned  to  the  links.  The  model  can  be 
easily  changed  to  allow  algorithms  with  link  memory. 

At  a  given  time  t,  each  link  j  has  a  control  value  P^t) 

and  a  flow  fj(t),  and  each  session  i  has  rate  r*(t).  We 
assume  that  the  system  begins  operation  at  t=0,  that  is, 

Pj(t)=0  and  ri(t)=0  for  t<0.  At  t=0,  the  links  and  sessions 
choose  some  arbitrary  initial  controls  Pj(0)  and  rates  r^O). 

For  each  link  j,  session  i£^  has  an  apparent  rate 

rij(t),  which  is  the  most  recent  rate  communicated  to  link  j 
by  session  i  at  time  t.  The  apparent  rate  rij(t)  may  differ 
from  r^(t)  because  of  communication  delays.  Hence  we  have 

rijCt)sri(t-dij(t)),  (3.7) 

where  d ^ ^ C t )  is  the  delay  described  above.  We  will  call 
dij(t)  the  propagation  delay.  With  this  definition  of  rate, 
flow  is  defined  as 


(3.8) 


Tij(t) 


When  the  system  has  been  running  long  enough  for  a 
session  i  to  have  received  feedback  from  each  of  the  links  in 
its  path,  the  rate  ri(t)  at  which  a  session  sends  is  chosen  as 
the  minimum  of  the  control  values  of  the  links.  Again, 
because  of  communication  delays,  the  most  recent  values  of  the 
control  for  link  j  known  by  session  i  may  not  be  current. 


Hence, 


i(t )  =  | min  PjCt-Dij(t))  when  t>Dij(t) 

Jr i(0)  when  0<t<Djj(t)  for  some  j£j£i 


when  t<0. 


(3.9) 


We  will  call  D.^t)  the  feedback  delay. 


We  make  two  assumptions  about  the  processes  d.  .(t)  an(j 

*  J 


Dij(t).  First,  for  tf^t2 
t1-dij(ti )4t2-dij(t2) 


(3.10.1) 


t1-Dij(ti )it2-Dij(t2).  (3.10.2) 

This  guarantees  that  new  information  is  not  replaced  by  old. 


Second,  for  any  times  t ^ j 0  and  T^0,  there  must  exist 


tijIitijO  and  such  that 


(3.11.1) 
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This  assumption  guarantees  that  the  links  and  sessions  will 
never  stop  receiving  new  information  about  the  rates  and 
controls  current  in  the  network.  That  is,  if  a  link  j  has 

control  Pj(t)  at  some  time  t,  then  each  of  its  sessions  will 
eventually  learn  the  value  of  the  link's  control  at  t  or  some 
later  time.  Similarly,  the  apparent  rate  of  session  i  on  link 
j  must  eventually  reflect  the  changes  in  the  session's  input 
rate  ri(t). 


Together,  these  two  assumptions  guarantee  that  the  system 
eventually  changes  from  its  initial  conditions.  If  we  let 
Tij°=0,  then  (3.11.2)  guarantees  the  existence  of  a  time  tQ>o 
such  that  Dij(to)itQ.  Furthermore,  by  (3.10.2), 

°<tO-Dij(to) 

<t-Dij(t)  (3.12) 
for  all  t>tQ.  Hence  D^j(t)it  for  t2tg.  Thus,  for  large 
enough  t,  (3.9)  becomes 


ri(t)=  min  p1(t-Di1(t)) 
and  (3.7)  becomes 

rij(t)=  min  pk(t-  dij (t )”Dik(t-dij(t) ) ) . 


(3.13) 


(3. 14) 


The  controls  for  the  links  are  updated  as  follows.  For 
each  link  j,  we  are  given  a  monotonic  increasing  sequence 
(tjO)  for  n^O,  with  t^O^O.  We  define  pjn=pj (t jn)  and 
fjn*fj(tjn).  We  assume  that  Pj(t)=Pj(0)=Pj°  for  0<titj°. 


Then  tjn  is  the  time  of  (n+1)st  control  update  for  link  j,  and 
Pjn+1_Pj(^nf  tr^j(tjn)  •i^-J-  j}  »Cj»Wj)  (3.15) 

where  Pj  iS  the  control  update  function  for  link  j, 

Pjn=^Pjn»Pjn“1 » • • • iPj®)t  cj  and  Wj  are  the  capacity  and  weight 
of  link  j,  respectively.  The  control  for  link  j  at  time  t, 
for  tjn-1  <t.£t jii,  is  Pj(t)  =  pjn*  See  Figure  3.1. 


Note  that  Hayden's  algorithm  is  a  special  case  of  this 


model,  obtained  by  letting  dij(t)sDij(t)s0  for  all  t,  and 
t^nm  for  all  n,  for  each  j,  and  defining 


Pj(£jn»  *rij(tjn)  :iGXj)  ,Cj,Wj)  =  pjn+(cj-  rij(tjn^/wj 


=P4n*(cj-fjn>/wj. 


(3.16) 


Since,  in  general,  djj(t)jfO  and  Dij(t)^0,  and  because  it 
is  difficult  to  synchronize  links  such  that  t.nsn,  it  is 

J 

useful  to  ask  for  what  values  of  d^t),  D^Ct)  and  tjn  we  can 
expect  the  rates  r^t)  to  converge  to  some  desired  set  of 
values. 


3.3  Jipdatfi 


FI ow  Control 


In  this  section  we  consider  asynchronous  flow  control 

algorithms  such  that  the  system  can  select  the  times  t-n 

3 

according  to  some  established  criteria.  Such  a  set  of 
criteria  is  called  an  update  protocol.  We  examine  some 


possible  update  protocols,  in  conjunction  with  different 


Wf77 


update  functions 


Let  us  consider  an  algorithm  using  Hayden's  control 
update  function,  but  with  dij(t)/0,  Dij(t)/0  and  tjn/n.  If 
upper  bounds  are  known  for  dij(t)  and  Dij(t),  it  might  be 
possible  to  eliminate  the  effects  of  these  delays  on  the 
controls  and  rates  by  waiting  "long  enough"  between  control 
updates.  That  is,  we  would  like  to  choose  t,n  such  that 

J 

rij(tjn)  =  ri(tjn)=  min  PicUjn>*  (3.17) 

In  this  case,  the  rates  and  controls  would  be  identical  to 
those  for  a  system  in  which  di j(t)=Dij(t)=0. 

Since  the  proof  of  convergence  for  Hayden's  algorithm 
relies  very  little  on  the  synchronous  properties  of  his 
algorithm,  his  proof  is  easily  modified  to  show  convergence 
for  an  asynchronous  algorithm  where  (3.17)  holds. 

Unfortunately,  the  condition  given  in  (3.17)  is  difficult 
to  meet,  since  it  requires  that 

*  min  Pk(tjn-dlj(tjn)-Dik(t  n.d  (t  n») 

s  min  Pk(tsn)*  (3.18) 

Necessary  conditions  for  (3.18)  to  hold  are  complicated,  but 
it  is  clearly  sufficient  that 

^k(t jn ^sPk(t  jn“di  j  ( ^"Dik(t  jn“di  j ( j*1  ^  ^ 


(3.19) 


(3.20) 


for  each  k  If  we  choose  tjn  such  that 

then  (3.19)  is  satisfied  for  ksj.  But  to  guarantee  (3.19)  for 
krfj,  it  is  necessary  that  there  be  no  updates  to  the  control 
for  link  k  between  the  time  that  session  i  learns  the  latest 
value  of  p^n  and  the  time  that  the  flow  on  link  j  reflects  any 
possible  change  in  r^t)  dUe  to  a  change  in  pkn.  This  could 
be  accomplished  by  making  the  links  and  sessions  update  during 
alternating  intervals.  That  is,  during  a  link  update  interval 
all  links  will  update  their  controls,  while  sessions  may  not 
update  their  rates.  The  reverse  is  true  for  a  session  update 
interval.  This  scheme  has  the  disadvantages  of  being  slow  and 
requiring  additional  communications  to  inform  sessions  and 
links  of  the  ends  of  the  update  intervals. 

Another  approach  would  be  to  perform  updates  only  when 
all  the  sessions  on  a  link  are  either  aware  of  the  link's 
current  control,  or  are  being  controlled  by  another  link  whose 
control  is  smaller.  That  is,  choose  t.n  such  that 

J 

(3.21) 

If  the  links  are  able  to  observe  the  rates  of  their  individual 
sessions,  then  the  condition  of  (3.21)  is  easy  to  enforce, 
provided  that  a  sequence  of  times  {t^}  satisfying  (3.21) 
actually  exists.  Fortunately,  the  delay  conditions  (3.10)  and 
(3.11)  guarantee  the  existence  of  such  a  sequence. 


We  want  to  show  that  for  each  link  j  and  any  time  t1 f 

there  exists  t2^tj  such  that  rij^t2^Pj^t2^»  for  each  i€,£j. 
We  will  need  the  following  theorem: 

Theorem  U..  If  d^t)  and  DAj(t)  satisfy  (3.10)  and  (3.11), 
then  for  any  t®  there  exists  t^t^  such  that  for  all  t2t^ , 

t-dij(t)-Dik(t-dij(t))it°*  (3.22) 

for  all  jeg,  i^ljf  k£j?it 

The  proof  of  Theorem  3.1  is  given  in  Appendix  A. 

Now  suppose  that  no  such  t2  existed.  Then  no  link 

updates  could  take  place  after  tj  #  jjow  pj(t^)  =  pjn  for  some  n, 

and  so  Pj(t)=pjn  for  al*  t>tj .  But  by  Theorem  3.1,  there  must 
exist  t*>t1  such  that  for  all  tit' 

t-dij(t)-Dij(t-dij(t)))2t1 .  (3.23) 

By  (3.14),  for  large  enough  t, 

rij(t)=  min  Pk(t-dij(t)-Dik(t-dij(t))) 

<  Pj(t-dij(t)-DiJ(t-dij(t))) 
s  PjCti) 

s  Pj(t),  (3.24) 

which  is  a  contradiction.  Hence,  an  infinite  sequence  of 

times  {t.jn}  must  exist  such  that  (3.21)  is  satisfied. 

J 


Unfortunately,  it  is  not  possible  to  guarantee 
convergence  of  the  rates  for  a  network  using  the  control 


update  function  in  (3.16)  and  the  update  protocol  in  (3.21). 
A  counter  example  is  easily  constructed  using  the  following 


>  ■ 
i  .'»■ 


argument.  Suppose  that  for  some  link  j  at  time  t,0,  (3.21)  is 

J 

satisfied  and  fjn<Cj.  Then  pjn+1  =  Pjn+(c j-f jn)/Wj  >  Pjn* 
Now  at  t^+e,  p.n+1>p.n,  and  if  e  is  taken  small  enough,  the 
rates  will  not  have  had  sufficient  time  to  change  and 
rij(tjn+e)=  rij(tjn)<  Pjn<  Pjn+^ .  Hence  the  link  may  update 
again  "immediately w.  By  updating  the  control  arbitrarily 
often,  the  control  can  be  increased  to  any  desired  value. 


An  obvious  approach  to  fixing  this  problem  is  to  require 

that 

t  n+1  -  t ,ri  2  x  (3.25) 

J  J 

for  some  positive  x,  for  all  jCjf,  for  all  n.  Another 
approach  would  be  to  put  an  upper  bound  on  the  value  of  p^. 
While  it  seems  likely  that  either  of  these  conditions  would 
guarantee  convergence  of  the  rates,  we  have  not  been  able  to 
prove  this. 


3.4  An  Asynchronous  flow  Control  Algorithm 


Fortunately,  it  is  possible  to  prove  convergence 
rates  for  a  network  using  this  update  function: 

h”*1  *  rij‘tjn>  *  (crfjn>/wj 

along  with  the  update  protocol  given  in  (3.21),  but 


of  the 


(3.26) 

without 


any  additional  requirements 


The  new  update  function  has  the  further  advantage  over 
Hayden's  that  it  guarantees  convergence  of  the  controls  as 
well  as  the  rates.  For  Hayden's  algorithm  the  controls  may 
diverge,  since  the  algorithm  finds  the  fair  control  allocation 
over  the  set  defined  by 

Vj-  (3-27-1) 
Thus,  if  a  link  controls  none  of  its  session,  the  fair  control 

for  that  link  is  infinity.  Our  algorithm  finds  the  fair 

allocation  over  the  set  defined  by 

Si  maX  min  Pk)icJ  (3.27.2) 

It  is  easy  to  see  that  the  fair  rate  vector  produced  by  either 

(3.27.1)  or  (3.27.2)  is  the  same,  for  if  a  link  j  controls  any 

of  its  sessions,  Pj=max  min  p^,  and  the  two  conditions  are 
equivalent.  Otherwise,  it  does  not  matter  what  control  link  j 
is  assigned,  since  it  does  not  affect  the  rates. 

We  can  find  the  fair  control  vector  over  the  set  defined 
by  (3.27.2)  by  a  global  procedure  almost  identical  to 
Hayden's.  We  begin  by  finding  the  bottleneck  link  j, 
assigning  it  Pj*=Cj/Wj  and  assigning  its  sessions  rates  equal 
to  Pj«.  we  then  form  a  reduced  network  by  deleting  that  link 
and  its  sessions  from  the  original  network,  and  reducing  the 
capacity  of  the  remaining  links  by  the  appropriate  amount.  We 
repeat  this  procedure,  finding  the  new  bottleneck  link  and 
reducing  the  network,  until  all  links  have  controls  assigned. 
However,  we  may  eventually  find  that  our  reduced  network 


contains  a  link  that  has  no  users.  The  fair  allocation  over 
(3.27.1)  sets  the  control  for  such  a  link  to  infinity,  whereas 
the  fair  allocation  over  (3.27.2)  sets  it  to 

Pj«s  max  min  Pk+(cj- ^  «*i«)/Wj.  (3.27.3) 

We  may  interpret  (3.26)  as  follows.  Hayden's  algorithm 
fails  to  produce  controls  that  converge  for  two  reasons.  In 
the  absence  of  any  update  protocol  a  link  may  update  its 
control  before  all  of  its  sessions  have  learned  the  current 
control  value,  thereby  producing  oscillations  in  the  control. 
Furthermore,  when  a  link  controls  none  of  its  sessions,  it 
increases  its  control  without  bound.  The  update  protocol 
overcomes  the  first  problem  by  letting  the  link  pretend  that 
all  its  sessions  know  its  most  recent  value.  The  link  simply 
assumes  that  those  sessions  that  are  sending  at  a  lower  rate 
are  being  controlled  by  some  other  link.  The  update  function 
overcomes  the  second  problem  by  making  the  link  pretend  that 
its  current  control  is  actually  the  rate  of  its  fastest 
session.  Thus,  whenever  an  update  takes  place,  the  link  is 
effectively  controlling  at  least  one  of  its  sessions. 

Next  we  prove  that  the  algorithm  given  by  (3.21)  and 
(3.26)  produces  input  rates  and  link  controls  that  converge  to 
the  desired  values.  We  give  the  following  theorem: 

Theorem  3.2.  Let  7£=(J,2£)  be  any  network.  Let  Pj(t), 
fj(t),  ri(t),  i*jj(t),  pjn  and  fjn  *>e  given  according  to  the 
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asynchronous  flow  control  model,  where  the  control  update 

function  is  given  by  (3.26),  the  control  update  times  tjn 

satisfy  (3.21),  and  the  delay  functions  djj(t)  and  Djj(t) 

satisfy  (3.10)  and  (3.11).  Then,  for  any  initial  control 

vector  J&0  and  initial  rate  vector  £®,  lim  £(t)s£*  and 

lim  £(t)=E*,  where  is  the  fair  allocation  over  the  set 

A. 

defined  by  (3.27.2)  and  £*  is  fair  over  the  set  defined  by 

(1.2). 

In  order  to  prove  Theorem  3.2,  we  will  need  the 

following  lemma.  The  lemma  says  that  the  control  for  a 

"bottle-neck"  link  converges  to  Pj*f  in  a  network  where 

the  flows  are  perturbed  from  their  true  values,  provided  that 

the  perturbations  eventually  go  to  zero.  This  implies  that 

the  control  for  of  a  network  with  no  perturbations  must 

converge,  and  the  rates  of  its  sessions  must  also  converge. 

To  the  other  links,  this  network  is  indistinguishable  from  a 

reduced  network  obtained  by  deleting  link  j  and  all  its 

sessions  from  the  original  network,  decreasing  the  capacity  of 

each  remaining  link  k  by  ri*,  and  perturbing  the  flow 

fk(t)  by  .  .  (ri*-rj.(t ) ) .  Since  the  perturbations  in  the 

A.fci'Vi  in¬ 
flows  go  to  zero,  by  the  lemma,  the  controls  for  the 

bottle-neck  links  of  the  reduced  network  must  also  converge. 

By  induction,  we  may  then  show  that  the  controls  for  all  the 

links  converge. 

Lemma  3.2.  Let  (<$  )  be  any  network.  For  each  j ,  let 


A  jy 

Wj  be  some  constant  satisfying  Wj^Wj,  let  {ejnJ  be  some 


sequence  that  converges  to  0,  and  let  { t j n }  be  a  sequence  that 


satisfies  the  conditions  given  below.  Let  £(t)=.Q  and  r(t)=Q. 


for  t<0  and  choose  any  initial  vectors  fi(0)  and  £(0).  Let 


Pj(t)=Pj(0)=Pj°  for  0<titj0  and  define 


Pjn.’.  max  rlj(tjn).(Cj.f jO-ejU/Wj 


jAL. 


(3.28) 


A  A  ° 

where  f^n,  rj.(t)  are  defined  as  in  (3.7)  -  (3.9),  and 

Pj(t)spj(tjn>  for  fcjn“^ <t<t jH.  Assume  that  and  Dij(t) 


satisfy  (3.10)  and  (3.11).  Further  suppose  that  the  sequence 
{t^n}  is  chosen  so  that 

J 

^ij(tjn)iPj^tjn)  Vj^Yiefj,  n20, 

ice  {e j n }  is 

V#ii,  Vj ^ 


and  that  the  sequence  (efn)  is  chosen  so  that 

J 


Pjn>° 


(3.29.1) 

(3.29.2) 


Let  pU  min  c./w-  Define  X1  and  in  the  obvious 

^  J  J 


manner.  Then  lim  'p.i(t)  =  p1  and  lim  rii(t)  =  p1  Tor  all 

J  J 

i^Jtl.  Furthermore, 

(3.29.3) 


A 


and 


lim  inf 

J  J 


lim  inf  ri1(t)^  min  zk* 
for  all  j€jl,  ieljf  where 

zJ*=PU(oj/Wj-P1)t'j/C2Wj). 


(3.29.4) 


(3.29.5) 


Proof  of  Lemma  3.2, 


We  begin  by  showing  that  lim  inf  P^tJ^Zi*  and 

«o  J 


•fc  *-  N  .N  , 


*  ■  .  .*  /  .«  *  .*  .*  .*  .*  •  *  /  -  * 

V  V  *V* 
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lim  inf  min  zt,*.  We  do  this  by  constructing  monotonic 

increasing  sequences  of  lower  bounds  Zjn-*,Zj»  for  eaCh  link  j, 
and  a  single  monotonic  increasing  sequence  of  times  {Tn},  such 
that  for  all  t^Tn,  £j(t)>Zjn  and  r\jj(t)>  min  zkn,  for  each 
We  invoke  the  following  lemma. 

Lfitnma  3.2.1 .  For  each  j£j£_,  let  ZjOsO  and  define 

ZjfH-1  s  (1-Wj/2Wj)  min  zkn  ♦  cj/2Wj.  (3.30) 

Then  zjn-^Zj*  an(j  zjn<cj/Wj  for  all  n. 

The  proof  is  given  in  Appendix  B. 


Let  ZjH  be  defined  as  in  (3.30).  We  now  show  how  to 
construct  the  sequence  {Tn>.  By  Theorem  3.1,  we  may  choose  T° 
so  that  for  all  tiT° 

min[tft-dij(t)-Dik(t-dij(t))]>  max  t!°  (3. 3D 

for  all  JC*,  k6^i#  Now  if  tiT°»  then  Pj(t)spjn  for 

some  n21.  Hence,  by  (3.29.2),  <Pj(t)>Zj°s0*  Furthermore, 

0  a 

rij(t)s  min  pk(t-dij(t)-Dik(t-dij(t) ) ) 

!<«*{ 

i  Bin  z„0.  (3.32) 

Now  suppose  there  exists  Tn  such  that  for  all  t2Tn, 
Pj(t)>Zjn  and  »*ij(t)>  rain  zkn,  for  all  ,  if  i-j.  since 
Zjn<cj/Wj,  we  may  choose  M  large  enough  such  that,  for  all 
m>M,  1  e jm | <(Cj_Wjz jn)/2  and  tjm>Tn,  for  each  Now  choose 


(3.33) 


Tn+1  such  that,  for  all  t2Tn+* , 

min[t, t-dij(t)_Dil{(t-dij(t ) ) ]>  max  tiM* 


for  all  i <£J.Jf  k  If  WTn+1.  then  Pj  C  t  )s£jm+1  for 

some  m2M.  Hence, 

V*1'  ■“  fijttj*)  ♦  <Cj-  21 

>  ““  *  t«j-Wj  ... 

^  6ji 

*  (1‘Wj/Wj)  max  rij(tjm)  +  (°j  -  ejm^wj 

>  C-Wj/Wj)  min  Zkn+cj/Wj-(c j-WjZ jn)/(2W. ) 

=  d-Wj/2Wj)  min  Zkn  ♦  cj/2Wj 


r  Zjn+1 


(3.34) 


Thus,  Pj(t)>zjn+1 .  Furthermore 

rij(t)=  min  pVk(t-dij(t)-Dik(t-dij(t))) 

>  ®in  zkn+1.  (3.35) 

By  induction,  we  have  constructed  the  sequence  (Tn)  as 
desired. 


Next  we  show  that,  for  each  e>0,  there  exists  T  such  that 
for  all  t£T ,  PjUXp^We  and  ^^(tXpUWe  for  all  J  , 
i  ,  where  W=  max  W.  Since  we  already  have 

lim  inf  P1(t)ip1  and  lim  inf  rii(t)^p1  for  iGU*  this 

^oo  J  7t  «0  J 

will  complete  the  proof  that  pj(t)_yp1  and  fijCtl-^p1  for 

jc^l,  1&|1. 


Let  j  be  a  link  in  ^  and  let  i  be  a  session  on  j. 

Choose  N  large  enough  that  r^j(tjn)>P^“®  and  ejn<(W-Wj+i  )e/2 

for  all  n>N.  Suppose  that  max  r\  x(t in)2p1+We  for  some  n2N. 

J  3 

Then  o 

?jn2(Wj-i)(p1-e)+  max 

k. 


V 

>(Wj_i )  (pl-eJ+pUwe 
=WjpU(W-Wj+1  )e. 


(3.36) 


Hence, 

A 


-  yy  _  A 

p,n+1<  max  rij(tjn)+(cj-(Wjp1+(w"Wj+1 )e)-ejn)/wj 

A 

<Pjn-(W-Wj+1)e/Wj-ejn/Wj 
<p.n_(w_wj+1 )e/(2Wj) 


<Pjn» 


(3.37) 

where  the  first  step  follows  from  the  update  protocol  (3.29.1) 
and  the  fact  that  P1=cj/Wj  for  So  as  long  as 

max  rij(tjn)>P1+W«*  Pjn  must  decrease  by  at  least 

)e/(2W.)  until  max  ?iJ(tjn><P,'fMe  for  s0“e  ‘j”' 


Now  suppose  max  rjj(t jn)<p^+We.  Then 

^  A 


p^n+Umax  ^ij(tjn>+[cj-(Wj-1 ) (p1"e,”naJ  rij(tjn)"ejn;i/Wj 

^  C*j  ^ 

fd-l/Wj)  max  rij(tjn)  ♦  <P1+<wj-!  )e-ejn)/WJ 


/CC'X 


<( 1-1/Wj ) (pl+We)  +  (p1+(Wj_i )e+(W-Wj+1 )e/2)/Wj 
<n-1/Wj)(pUvTe)  +  (p1+(W+Wj_i)e/2)/Wj 
<(1-1/WvJ)(p1+We)  +  (pUWe)/ffj 
spl+We. 


(3.38) 


/V  1  . 

So  once  max  r^(t<n)  falls  below  p'+We,  so  does  p,n+if  and 

•*  ,  — 

hence  both  must  remain  below  p'+We. 


So  for  each  e>0,  there  exists  N  such  that  for  all  n2N 
p1-e<pj  (t  jn)<p1+We  and  p1-e<rij(tjn)<P1+We  for  all  , 
i£ .  Hence,  there  must  also  exist  T  such  that  for  all  t^T, 
p1-e<Pj(t)<p‘l+We  and  p1-e<r^j (t)<p1+We  for  all  j^X1.  iC-V  ■ 
This  completes  the  proof  of  Lemma  3.2. 

We  are  now  ready  to  prove  Theorem  3.2. 


Proof.  Partition  the  links  in  into  sets  ££  1 » 

JC2*  •  •  •  »*CN  with  the  property  that  for  each  set  there 
exists  pk  such  that  Pj»spk  for  each  and  for 

Define  as  the  set  of  sessions  i  such  that 
Number  the  sets  so  that  pXp2<...<pL.  The  proof  is  by 
induction  on  the  sets  of  the  partition.  First  we  show  that 
the  sequences  (Pj Cn) }  for  each  j  in  and  (r^n)}  for  each  i 
in  jjL  1  converge  to  p^.  Then  we  show  that  if  the  sequences 
tPj (n ) }  for  j  in  Xk  and  f**i(n>}  for  i  in  converge  to  pk 
for  all  k<K,  then  the  sequences  {p.(n)J  for  j  in  xLK+1  and 
(r^n)}  for  i  in  converge  to  pK+1 . 

In  order  to  prove  the  induction  step,  we  will  also  need 


to  show  that  if 

lim  inf  rij(t)>pl 
X"-*  oo  J 


(3.39) 


s'y.1. 
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f0r  j  then  (3.39)  holds  for  j€-  L'j£h, 

4~>*+) 

First  note  that  p1=  min  p.#_  min  Ci/Wi.  Now.  let 
p(0)=p(0)  and  £(0)=r(0).  For  each  j(=£,  let  Wj=v*j  an(j  iet 
ejn=0  for  all  n.  Then  Pj(t)=pj(t)  and  rijUJsrfj  ( t)  for  all 
t.  Therefore,  by  the  lemma,  lim  P-j(t)  =  p1sPi*  and 

lim  rij(t)sp1*rie  for  j^X1,  ^A-1  * 

We  must  also  show  that  (3.39)  holds  for  K=1.  Now  for 
i^C/J.1,  ri«>p1  and  hence  k^£1  for  each  k^Xi.  Therefore 

,t>±  * 

lim  inf  ri-(t)  2  min  zk« 

A  h 

>  pi,  (3.40.1) 

since,  by  definition,  zk«>p1  for  k^X?* 

Next  we  show  that  if  the  controls  and  rates  converge  to 
pk  for  the  links  and  sessions  in  and  ^Lk,  k^K,  and  (3.39) 
holds  for  K,  then  the  controls  and  rates  converge  to  pK+1  for 
the  links  and  sessions  in  ^£^+1  and  J^JC+1  f  and  (3.39)  holds 
for  K+1 . 

We  begin  by  defining  a  new  network  ^  '*(«!? )  where 

y*«  U  o£  and  X  ’s  (j  ^  •  Assign  the  links  in  Y  1 

^  Jk>K  ^  A>K 

capacities  c,»sC._  Z. ,  n*.  Let  the  weights  of  the  links  in 
X’  be  denoted  by  Vy.  Define  pj'(t),  fj'(t),  r^U), 

rij*(t),  Pjni  and  f j n *  in  the  obvious  manner.  Note  that 


pK+1s  min  Cji/yjfsp1'.  Define  X 1 '  and  ’  in  the  obvious 
mannerV  Note  that  V  1  »SJ^K+1  and 

For  each  either  W^.-o,  0r  Wj*>0.  If  Wj'sO,  then 

each  of  link  j‘s  sessions  is  in  for  some  k£K.  So,  by  the 

induction  hypothesis, 

lim  Pjn*'=  lin>  Cmax  r  ^tcj.  J  rij(tjn)/Wj] 

1-*~  LtS:  ; tii 


(3.40.2) 


s  max  min  Pk*+(cj-  ri»)/Wj 

J  < L  *4 

=p.»  « 


The  last  step  follows  from  the  construction  of  the  fair 
control  vector  (3.27.3). 


For  each  such  that  Wj.x,,  let'wj'rWj  and  define 


(3.41) 


V  „/<rij(tjn>-ri*) 

+Wj(  max  rij(tjn)  -  max  rij(tjn)). 

if  ^  j=  i-  j',  the  sum  is  empty  and  e 


e  n=0. 
eJ 


Pj  » (O)spj  '  (O)spj(O)  for  each  and  r^  (0)  =  ri'  (0)  =  ri(0)  for 

each  i  6  J_' •  Define  Pj  •  (t),  fj«(t),  r^U),  r^'U),  p^0'  and 

.A  A 

fjn'  as  described  in  Lemma  3.2.  We  will  show  that  Pjn'sPjn 

and  >^ijn,  =  rijn  for  all  n.  We  will  then  show  that  (3.29.1)  and 

(3.29.2)  are  satisfied  and  that  e.n-^0.  We  may  then  apply  the 

J 

lemma. 


We  show  that  Pjn,=Pjn  and  ^ijn,srijn  by  induction.  First 
note  that  if  j  is  the  first  link  to  perform  a  control  update, 


then  for  t<tjO,  Pk'(t)=pk(t)  and  rik'  (t)=rik(t)  for  each 

keX'- 


Now  suppose  there  exists  some  t^n  such  that,  for  t^t^n, 

Pj'(t)spj(t)  and  r^ij '  (t)  =  rij(t)  for  each  j^X'»  Let 

t1®  be  the  time  of  the  next  control  update  after  t^n.  For 

j*k,  Pj'Cti1”)  =  Pj'(tkn)  s  Pj(tkn)  =  Pj(tim>*  Furthermore, 

for  j=k, 

A  .  A  « 

Pk.(tin,)  =  Pkn+1’ 

^  Si 

=  max  rik' (tkn)+<ck*-fkn,“ekn)/wk‘ 

<fe5K< 

•  ma;  rik(tkn)*<ck-fk,1)/l'k 

jit  Jk 

Spkn*1 

•  PkCti"1).  (3.«2> 

The  second  step  follows  from  the  defintion  of  ekn.  Hence, 

pj  ’  (t)spj(t)  for  each  j C-£',  when  titi®.  This  last  remark 
also  implies  that  ^1j»(t)»r1j(t)  for  jC-£\  i^Xj’t  when 
tit^m. 


Because  riJ.(t)srij(t), 

PjO+1 »spjn+1 


(3.29.1)  is  satisfied. 


-  “aj! 1  rij(ton)  +  (cj-Wj(  max  rij(tjn) >/Wj 

A£&j  '  a.  G  i ' 

SCJ/WJ  4 


>0, 


(3.29*1)  is  also  satisfied. 


Since 


(3.43) 


Now  we  show  that  e.n.^0.  By  the  induction  hypothesis 

J 

rij(t)-ri«^0  for  ' »  i^J’jN^j1.  Also  by  hypothesis, 

rij(t)>p^  for  Hence,  there  exists  some 

N  such  that  for  n^N 

rij(tjn)>pK  (3.44) 

for  j^X'f  *  ' .  Therefore,  for  n2N, 

max  rii(tin)>pK 

j  c-2.'  J  d 

>  max  r,* 

=  lim  max  r .  .(t,0))  (3.45) 

for  j<^X’ •  Thus,  for  large  enough  n, 


max 


and 


n)>  max  rij(tjn) 

x>X- V-  / 

1 


(3.46) 


So 


"V  rij(tjn,= 

3  4 


(3.47) 


lim  [  max  r,  .(t  .n)- 
and  ejn-^>0,  as  desired. 


max  rij(tjn)i=0 


(3.48) 


Now,  by  Lemma  3.2  we  have  Pj'(t)-^?p1'  for  j^XJ  1  •  But 
'  (t)  =  Pj(t)  and  X1'  is  the  set  of  links  J  with  Pj»=pl  ,  =  PK+1  • 
So  Pj(t)-*pj«  for  j<^XK+1*  Similarly,  ri^t)-^ri  for  i^XK+1 . 


Finally,  to  complete  the  induction,  we  must  show  that 
(3.39)  holds  for  K+1 .  But  for  r,e>pK  and  hence, 
k  4  ^  X1  for  each  Therefore,  by  the  lemma, 
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■  v  ■  ^ 

’  ■*  \ 


*v  V'  \'  '.  *  -  -.*  v-  -.*  O  -  ' 


►•; 

I 

i."  • 


11m  inf  rlj(t)  =  lim  inf  r^'U) 

>  min  z^# 

>  pi  * 

s  pK+1 , 

as  desired.  This  completes  the  proof  of  Theorem  3.2. 


(3.49) 


3.5  Asynchronous  Algorithms  l&TL  More  General  feasible  Sets 


>>.- 

v.v 

v.‘ 


In  this  section  we  give  two  asynchronous  algorithms  that 
produce  fair  controls  and  rates  over  the  more  general  feasible 
sets  defined  by  (2.11).  The  algorithms  are  obtained  by 
modifying  the  synchronous  algorithm  given  by  (2.2)  in  much  the 
same  way  that  Hayden's  algorithm  is  modified  to  give  the 
asynchronous  algorithm  of  the  preceeding  section. 

First  we  change  the  asynchronous  system  model  so  that 
ri(t).Uln  b1"1(pJ(t-B1j(t)))  when  tlDjjU)  Vj  (Xl 

ri(0)  when  (Kt<Dij(t)  for  some 
0  when  t<0.  (3.50) 

The  new  update  protocol  requires  that  link  j  performs  updates 
only  when 

‘>l(r1J(tjn))<Pj(tjn>  Vieij.  (3.5,) 

Unfortunately,  when  the  flows  are  observed  this  update 
protocol  is  not  as  easy  to  implement  as  the  one  given  by 


«•  •»*. 


.V. 
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S  S 


(3.21).  Since  the  link  enforces  (3.51)  by  monitoring  its 
sessions'  rates,  it  must  know  how  to  calculate  ^i(rij(t))  for 
each  of  its  sessions.  But  this  defeats  the  intent  of  the 
original  algorithm  to  distribute  calculation  reasonably 
between  the  links  and  the  sessions.  When  flows  are 
communicated,  this  is  not  a  problem,  since  the  sessions  can 
simply  inform  the  links  of  both  rij(t)  and  bi(rij(t)). 

The  update  function  (3.26)  was  obtained  by  replacing  all 
occurences  of  p,n  in  (1.8.1)  with  max  r.-^ti11).  Hence, 
(2.2.1)  suggests  the  new  update  function  J 

pjn+1 s( 1_a jn)  max  bi(rij(tjn) >+ajn*j(cj-f ^n)  (3.52.1) 

where 

ajn=Aj(max  bi(i*ij(tjn)) )  (3.52.2) 

*  6^ 

where  Aj(.)  is  some  appropriately  chosen  function.  This  new 
update  function  does  produce  an  algorithm  that  behaves  as 
desired,  but  in  fact,  with  the  update  protocol,  it  is  not 
necessary  to  modify  the  update  function  of  (2.2.1)  at  all! 

To  see  why  this  is  so,  consider  the  case  where  gj(x)=x, 

bi(x)  =  x  and  ajn=1/(Wj+i)  for  each  ,  i£  J_.  Then  (2.2.1) 
becomes 

pjn+1=pjn+(cj-f jn"pjn)/(Wj+1  >•  (3.53) 

This  algorithm  gives  Jaffe's  fair  rate  vector  [10].  We 

interpret  this  update  function  as  follows.  Consider  Hayden's 
algorithm  in  the  case  where  each  link  j  has  a  "phantom" 
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session  that  experiences  no  feedback  or  propagation  delays, 
and  whose  path  consists  only  of  link  j.  Let  fjn  denote  the 
aggregate  flow  of  all  other  sessions  and  let  t>e  the  number 
of  all  other  sessions  on  j.  Then  at  all  times,  each  link  is 
controlling  at  least  one  of  its  sessions.  But  the  reason  for 
replacing  p^n  in  (1.8.1)  with  max  r 
that  each  link  could  pretend  to  be  controlling  one  of  its 
sessions,  and  thereby  prevent  unlimited  increases  in  its 
control  value.  Since  each  link  is  already  controlling  a 
"phantom"  session,  it  is  not  necessary  to  modify  the  update 
equation  at  all. 


in  was  so 


We  give  the  following  theorem. 


Theorem  3.3.  Let  7*  s(  .J  )  be  any  network.  Let  gj(-)  and 


*>*(•)  satisfy  Assumption  1.1.  Let  fj(t),  rjU),  and  rij(t)  be 
given  according  to  (3.7),  (3.8)  and  (3.50).  Let  Pjn=Pj(tjn) 


and  f* jn=  f j  ( t j n 5  where  the  control 


update  times  tjn  satisfy 


(3.51),  and  the  delay  functions  ^(t)  and  Dij(t)  satisfy 


(3.10)  and  (3.11).  Let  the  control  update  function  be  given 
by  (2.2.1),  with  ajnsAj(pjn) •  Let  6j(x),  Aj(x),  Gj(x,c,S)  and 
Hj(x,c,S)  be  as  in  Theorem  2.2.  Then,  for  any  initial  control 

vector  and  initial  rate  vector  r°»  lim  jj(t)=£*  and 

✓f-S’oO 

lim  r(t)s£»,  where  and  £*  are  the  fair  allocations  over 
the  set  defined  by  (2.12). 


Before  proceeding  with  the  proof  of  this  theorem,  we 


prove  the  following  useful  lemma 


Lemma  3.3.  Let  f}  =  (£  ,££)  be  any  network.  Let  b^O,  8j(*) 

and  Aj(x)  pe  as  in  Theorem  2.2.  For  each  j,  let  tejnJ  be  any 
sequence  that  converges  to  0,  and  let  {t^}  be  a  sequence  that 
satisfies  the  conditions  given  below.  Let  j>(t)=jQ  and  r(t)=fi 

A  A 

for  t<0  and  choose  any  intial  vectors  p(0)  and  r(0).  Let 
^j(t)spj(0)=^j°  for  0<titj0  and  define 

Pjn+lad.^njp^n+^ng^c^n.ejn)  (3.54) 

/V  A 

where  fjn,  ( t ) ,  ri(t)  are  defined  as  in  C 3 - 7 ) *  (3.8)  and 

(3.50),  ^jnsAj(pjn),  and  £j(t)spj(tjn>  for  t^-la^n. 

Assume  that  dij(t)  and  Dij(t)  satisfy  (3.10)  and  (3.11). 

Further  suppose  that  the  sequence  {t.n}  is  chosen  so  that 

J 

bi<rij(tjn)KPj(tjn)  Vi^i-j,  n±o,  (3.55.1) 

and  that  the  sequence  { e j n }  is  chosen  so  that  Pjn  is  bounded 
below.  That  is, 


Pjn>zj° 

for  some  Zj°. 


Vn21,  Vj^, 


Let  pU  min  p.«.  Define  X 1 

manner.  Then  lim  p^(t)=D'  and 

J'W  V 

i<=\£J.  Furthermore, 
lim  inf  b^rjjCt) ).2  min  zk»,  for 

J  KexV 


(3.55.2) 


and  X.  1  in  the  obvious 
lim  bi(rii(t) )sp1  for  all 
lim  inf  Pi(t)2z  1*  and 
all  j<r^,  where 


Proof  of  Lemma  3-*_3. 


First, 


we 


will 


show 


and 


lim  inf  Pj(t)2zj« 
lim  inf  b1(r'..(t))^  min  z^*.  Then  we  show  that  for  each  e>0, 
there  exists  T  such  that  for  all  t>T,  Pj(t)<p1+Ej(e)  for  each 


j  ,  where 


Ej(e)=2(Gj(p1-e,cj+e,^ jj-p1).  (3.56) 
This  is  sufficient  to  show  the  convergence  of  the  controls, 
since  Xj-pl  for  j  Gjrl ,  and  because  Ej(.)  has  the  properties 
that  it  is  monotonically  non-decreasing,  and  E.j(o)=o.  The 
convergence  of  the  rates  follows  directly  from  the  convergence 
of  the  controls. 


We  show  lim  inf  P<(t)2zi*  and  lim  inf  bi(rii(t))2  min  zu* 
by  constructing  monotonic  increasing  sequences  of  lower  bounds 

2jn-?zj«  and  a  single  monotonic  increasing  sequence  of  times 
tTn } ,  such  that  for  all  tj£Tn,  Pj(t)>zjn  and 

bi(rlj(t))>  rain  zkn,  for  each  Let  Zj°  be  as  in 

(3.55.2)  and  define  Zjn  by 

zjn+1  s  Hj(  “in  2kn*cj-iejn|*  ^-j).  (3.57) 

ke£. 


First  we  show  how  to  construct  the  sequence  CTn } .  Then  we 
show  that  Zjn^Zj«< 


By  Theorem  3.1 »  we  may  choose  T°  such  that  for  all  tj>T° 

mintt, t-d^j(t)-Dij{(t-dij (t) ) 3>  max  t^®  (3.58) 

C  G  A 

for  all  k^Xi.  Now  if  tiT°*  then  Pj(t)=p'jn  for 

some  n21.  Hence,  by  (3.55.2),  Pj(t)>zj°.  Furthermore, 

bi(nij(t))s  min  Pjt(t-dij(t)-Dij<(t-dij(t))) 
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>  min  z  |0 . 
*€&  * 


(3.59) 


Now  suppose  there  exists  Tn  such  that  for  all  t2Tn, 

Pj(t)>zjn  and  bi(rij(t))>^min  zknt  for  all  i€£j. 
Choose  M  large  enough  such  that,  for  all  m^M,  lejmi<iejnl  and 
tj®>Tn,  for  each  Now  cboose  Tn+1  such  that,  for  all 
t^Tn+1 , 


min[t,t-dij(t).Dik(t.di.(t))]>  max  tlM. 


(3.60) 


for  all  If  t*Tn+1.  then  'Pj(t)='pjID+1  for 


some  m^M.  The  update  protocol  (3.55.1)  guarantees  that 
f  21  b.-l(p.ni). 

Hence,  we  have 


(3.61) 


Pj®+12(1-ajra)>Pjni+ajnigj(Cj.  bi”1  (p'jm)- i e jn  1 ) 

/.e-ij 

=  HjCj/.Cj-i./l.i-j) 

>  Hj(  min  zkn»Cj-!ejni»  i.j) 

=  2 jn+1 f 


(3.62) 


where  the  next  to  last  step  follows  from  (2.27).  Thus, 
a  i 

Pj(t)>zjn+1 •  Furthermore 

bi(rij(t))=  min  (pk(t-dij(t)-Dik(t-dij(t)) )) 

>  min  z. n+^ .  (3.63) 

By  induction,  we  have  constructed  the  sequence  {Tn}  as 


desired. 


Now  we  show  that  To  <j0  this  we  invoke  the 

following  theorem. 

Theorem  3.4.  Let  S  be  a  linear  space  with  norm  SI*  I!  such 
that  {x:  ISxll^c}  is  compact  for  all  c.  Let  f:S~9S  and 

fn:S-*S  be  functions  such  that  fn-^f  uniformly,  and  such  that 
ilf(x)  -  f  ( y )  I  S  <  I  I  x  -  yli  for  all  x,y£S.  Suppose  there 
exists  x*  such  that  x*=f(x«).  Define  xn+1=fn(xn).  Then 

xn^x#* 

The  proof  of  this  theorem  in  given  in  Appendix  C. 

Let  Fjn(z)sHj(inin  zk,Cj-ejn»i_j)  and 

Fj(i)=Hj(mi^n  zk,Cj,<^_j).  Let  Fn(*)s( • • • » Fjn(z) , . . . )  and 
F(z)s(...,Fj(2)f ...).  Since  e^n^O,  FnCz)-^F(i). 


Let  lizllsmax  |Zj|.  Then,  since  gj(*)  is  uniformly 
continuous,  for  any  £>0,  we  can  find  £  such  that 
I iFn(i)-F(z) | | 

=  max  IHj(min  zk,Cj-|ejni»>j)-Hj(min  zk,cj,J.j)| 

J  K  K 

smax  Aj(fflin 

1  k' 

•IGj(min  zk,Cj-|eJnl»^J)-Cj(min  zk,Cj,^j)J 
K 

<max  igj(Cj-|ei(n) I-  2  bi"1(minzk)) 

i  K 

-Mcj-'S,  ik,)| 


a, 

whenever  iej(n)j<£-.  Hence,  the  convergence  of  Fn(z)  to  F(z) 
is  uniform. 


Furthermore,  by  (2.27) 

I  |F(x)-F(y)  i  js  max  IHj(min  xk, cj  ,jL  j)-Hj(min  yk,Cj,Xj)l 
f  k  k 

<  max  imin  xk_min  ykj 

h  k  k 

=  imin  xk_min  yk| 

K  K 


<  max  !xk.yk| 


.  I  I  V  I  I 

-  I  I  I  • 


(3.65) 


Finally  we  show  that  z*  is  a  fixed  point  of  F(*)  Let  h 
satisfy  zh#s  min  zk».  Then 


‘h*s^h^  2h*»  ch*  h^ 


(3.66) 


where  Xh  iS  defined  by  (2.6).  Now  for  any  j,  Zj*2Zh*=xh  and 


zj«=Hj(Xh,Cj,ij) 


(3.67) 


Therefore 


Xh-Hj(Xh,Cj,£  j)iO 

sXj“Hj(Xj,cj,i  j),  (3.68) 

which  implies  xh-*Xj.  Hence  Xh=min  Xk=p1  and 


,*  \r~* 


*.  '■  •-  %  ~W 


FjUOj.Hjtp1. 


Hence,  by  Theorem  2.3,  2*. 


Next  we  show  that,  for  each  e>0,  there  exists  T  such  that 
for  all  t>X,  Pj(t)<p1*Ej(e)  and  MrijUlXp’.EjU)  for  all 

xyj. 


Let  j  be  a  link  in  and  let  i  be  a  session  on  j. 
Choose  N  large  enough  that  b^^jj  (tjn))-2P^“e  and  ! e j I <e  for 
all  n>N.  Now  suppose  that  Pjn>pUEj(e)  f0r  some  n^N.  Since 


fjn>^L  b^Upl-e), 


(3.69) 


we  have 


F^jn+ajn(gj(cj-ejn“SI  bj-1  (pl-e))-^**) 

a  ^  ^ 

lpjn+ajn(gj(cj+e-^  bi-^P’-eJJ-tP^E^e))) 

a  .  *  /e** 

=  pjn+ajn(GjCp1-e»cj+e,ip-(p1+E;J(e))) 

4jn-(1/2)Aj(^n,Ej(e) 

A 

<Pjn* 


(3.70) 


So  as  long  as  Pj^p^Ej^e),  "pjrw’1  wil1  be  less  than  Pjn» 
Furthermore,  the  amount  by  which  p^n  decreases  is  at  least 

(1/2)£j(e)  tnin  {Aj(x):x€  [p^+Ej(e),/pjM3)»  (3. 71) 

where  M  is  any  time  for  which  p^pl+E^e) .  The  minimum  in 
(3.71)  must  exist  because  Aj(x)  is  continuous  and  positive  for 
all  x.  Therefore,  'p‘in  must  eventually  be  less  than  pl+E^e). 

J  J 
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s  s 


Now  suppose  Pjn<P '+Fj (e) .  Then 

<(l4kJn)fjn^jnGj(p1+e,Cj.efi-.j) 
<(1-^Jn)(p1+Ej(e))+^jn(p1+(i/2)Ej(e)) 

<P1+EJ(e).  (3-72) 

Hence,  the  controls  p.n  converge  to  pi,  for 

J 

So  for  each  e>0,  there  exists  N  such  that  for  all  n2N 

pl-e<pj(tjn)<pl+Ej(e)  and  p1-e<bi(rij(tjn))<P1+Ej(e)  for  all 
Hence,  there  must  also  exist  T  such  that  for 

all  t>T,  P1-e<Pj(t)<p1+Ej(e)  and  p1*e<bi(r'ij(t) )<p1+Ej(e)  for 
all  This  completes  the  proof  of  Lemma  3.3. 

We  are  now  ready  to  prove  Theorem  3.3. 

Proof.  Partition  the  links  in  ^  as  described  in  the  proof  of 
Theorem  3.2.  The  proof  is  by  induction  on  the  sets  of  the 
partition,  and  is  analogous  to  the  proof  of  Theorem  3.2. 

Let  £(0)=£(0) ,  £(0)=r(0),  and  ejn=0  for  each  j6£,  for 

all  n.  Then  Pj(t)=Pj(t)  and  rij(t)srij(t)  for  all  t. 

Therefore,  by  the  lemma,  lim  P^(t)=p1sPi«  and 

J  J 

lim  bi(rij(t))=p1  =  ri«  for  i^1- 

Next  we  show  that  if  the  controls  and  rates  converge  to 
p*  for  the  links  and  sessions  in  ^ k  and  A_k,  k^K,  and  (3.72) 


for 


holds  for  K,  then  the  controls  and  rates  converge  to  pK+^ 
the  links  and  sessions  in  ^  K+1  and<^_^+^,  and  (3.72)  holds 
for  K+1 .  We  begin  by  defining  a  new  network  ft  '  =  (*£  f,}£j) 
exactly  as  described  in  the  proof  of  Theorem  3.2. 

For  each  let 

ejn=  ^EL  (rij(tin)-ri»).  (3.73) 

1 

If  j=Jj»,let  ejn=0«  By  the  induction  hypothesis,  e^h^O. 

Let  Pj  » (0)  =  pj  » ( 0)  =  pj(0)  for  each  and  r^'  (0)  =  ri‘  (0)=ri(0) 

for  each  1<£±<.  Define  p'j.(t),  fj.(t),  pjnl 

and  fjn>  as  described  in  Lemma  3.2.  By  an  argument  analogous 

to  that  given  in  the  proof  of  Theorem  3.2,  we  claim  that 
A  A 

Pjn,=Pjn  and  rijn,=rijn  for  all  n. 

Next  we  show  that  (3.54)  is  satisfied.  Because 
A 

rij(t)=rij(t),  (3.54.1)  is  satisfied. 

Let  zo<min(  min  p^iP1)  and  let  t,0  be  the  time  of  the 
first  link  update.  Then  Pj(t)>ZQ  for  t<ti°*  Now  suppose 
Pj(t)>ZQ  for  t<tjn*  Then 
PJn+1>Hj(pjn,cj,Xj) 

>Hj(Z0,Cj,  ij) 

>z0-  *  (3.74) 

The  last  step  follows  because  z0<p1<Xj.  Therefore,  Pj(t)>z0 
for  t<tkm,  where  tkm  is  the  time  of  the  next  link  update  after 
t^n.  So,  by  induction,  Pj(t)>z0  for  all  t.  Hence  (3.54.2)  is 
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satisfied. 


Now,  by  Lemma  3.2  we  have  p j  « ( t  p1  *  for  j^1».  But 
Pj'(t)  =  pj(t)  and  X.1*  is  the  set  of  j  with  Pj«=p1  »=pK+1 . 

So  Pj(t)^pj»  for  Similarly,  rij(t)--?ri«  for 

i£,^K+1#  This  completes  the  proof  of  Theorem  3.3. 

A  similar  theorem  can  be  proved  for  an  algorithm  using 
the  update  equation  (3.52).  The  proof  is  essentially  the  same 
as  for  Theorem  3.4,  except  that  in  the  lemma  we  define  p.n+1 

J 

by 

'Pjn+Ml-ajnHmax  bi( rjj ( t jn)  )+Ejn) 

*  <3-75) 

where  { E ^ n }  is  any  sequence  that  converges  to  0,  and  Ejn^0  for 
all  n. 

Note  that  Theorem  2.2  follows  directly  from  Theorem  3*4, 
by  letting  d^j(t)sD^j(t)sO  and  choosing  tjn=n  for  each 


Chapter  4 


General  Aay.nchroag.Ps  Distributed  Algorithms 

Bertsekas  and  others  [ 1 3]  have  developed  some  broadly 
applicable  results  pertaining  to  general  asynchronous 
distributed  algorithms.  In  this  chapter,  we  discuss  these 
results  and  show  how  they  can  be  modified  to  include  the 
algorithms  presented  in  Chapter  3. 

4.1  A  fifip.er.al  Convergence  Ih.e0X-fim 

In  this  section  we  present  Bertsekas'  main  result, 
slightly  reformulated  to  match  our  model. 

For  a  given  feasible  set  XCRn,  we  are  interested  in 
finding  an  element  of  the  solution  set  X*CX.  Such  an  element 
is  called  a  solution.  We  consider  a  system  in  which  a  network 


Of  N 

processors  iteratively  computes 

estimates  of 

the 

solution. 

Each  processor  i 

maintains  at 

all  times  t 

an 

estimate 

of  the  solution 

*i(t)6X,  and 

a  vector  of 

mi 

measurements  zi(t)s(z11 (t ), . . 

*  t * im .  ( t ) ) ,  as 

communicated 

to 

the  processor  by  the  network.  The  estimates  and  measurements 
are  updated  as  follows. 


Without  loss  of  generality,  we  index  the  times  at  which 
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the  events  of  interest  take  place  (such  as  a  processor 
updating  its  estimate  or  receiving  a  measurement)  by  an 


integer  variable  t.  We  also  assume  that  for  any  integer  t 

only  one  event  of  interest  occurs  in  the  system.  Suppose 

processor  i  receives  at  time  t  a  new  value  of  the  measurement 
A  A 

zij(t).  We  call  the  received  value  Z£j(t)  to  distinguish  it 
from  the  value  z^j(t)  currently  stored  by  the  processor.  Then 

zij(t+1 )szij(t) .  Furthermore,  the  processor  updates  its 
estimate  according  to 

„  A 

xi(t+1  )  =  Mij(xj.(t) , zii  (t), . . .  ,Zij(t), . . .  .z^U) )  (4.1 ) 

where  M.  .  (.)  is  a  given  function.  We  call  this  a  measurement 
update.  If  no  new  measurement  is  received  then 

zij(t+1  )  =  Zij(t).  Each  processor  also  updates  x^t)  from  time 
to  time,  according  to 

xi(t+1)=Fi(xi(t),zi(t))  (4.2) 

where  Fi(*)  is  a  given  function.  We  call  this  a 
self-generated  update.  When  no  new  measurement  is  received, 
and  the  processor  does  not  update  its  estimate,  xi(t+1 )=Xi(t ) . 
The  measurement  z^j(t)  received  by  processsor  i  at  time  t  is 
related  to  the  processor  estimates  x^  f  x£, . . . ,  Xfj  by 

zijCt)=GijCxi CTij^ (t)),...,x^(XijN(t))).  (4.3) 

(Bertsekas  also  includes  as  an  argument  to  Gjj(*)  a  random 
variable  Co  .  We  omit  this  for  simplicity,  since  the  systems 


we  are  studying  are  deterministic.)  We  make  the  following 
assumptions  about  the  times  at  which  measurements  are  received 
or  updates  take  place. 


Assumpt-.i nn  4._0.  For  all  1<i<N,  ixkiN,  Tjjk(t)lt. 


iL-L*  t-jltts,  then 

Tijk(tl)2Tijk<t2) 

for  all  1<i<N,  1_< 


(4.4) 


0,  there  exists 


4.2.  For  each  i  and  j,  and  any  t 
t^tQ  at  which  processor  i  receives  a  measurement  Zij(t-|)  of 
the  form  (4.3),  with  Ti>(t1)^t0  for  each  UkiN.  Also,  for 
each  i  and  any  tQ>  there  exists  t2>tg  at  which  processor  i 
updates  its  estimate  according  to  (4.2). 


Assumption  4.0  says  that  delays  must  be  positive,  that 
is,  we  cannot  predict  the  future.  We  call  this  the  causality 
assumption.  Assumption  4.1  is  essentially  equivalent  to 
(3. 1C).  Assumption  4.2  is  equivalent  to  the  result  of  Theorem 
3.1,  the  consequence  of  (3.10)  and  (3.11).  Bertsekas  calls  it 
the  continuing  update  assumption.  The  new  assumptions  are 
required  because  we  no  longer  assume  that  the  measurements 
zij(t)  are  updated  continuously. 

Bertsekas  gives  the  following  theorem. 


Theorem  4.1.  Let  Assumptions  4.0  -  4.2  hold.  Suppose  there 
exists  a  sequence  of  sets  { X ( k ) }  with  the  following 
properties: 
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X«CTX(k+1)ciX(k)C  .. .  cx 


(4.5.1) 


X»=  .0  X(k) . 

For  1<i<N,  and  k20,  define 

Zij(k)={Gij(x1,...,xN) !xh6X(k)  for  14h£N} 
^(kJstF^xi.zi)  |xi<s:X(k),Zi6Zi(k)} 
Zij(k)=(Gij(xi , . . . ,xu) Jxh^Xh(k)  for  1ih£N}, 


(4.5.2) 


(4.6.1) 

(4.6.2) 

(4.6.3) 


where  Z .  (k)sz^  j(k)x. .  .xZim.(k).  Let  Z±(k):Zi  j  (k)x. . .  xZ-^k) . 

Suppose  that  the  sets  X(k)  and  the  mappings  F^.),  G^j(») 
and  j ( • )  are  such  that,  for  all  i,  j,  k, 

Zi(k)dX(k)  (4.7.1) 
Mij(xifZi)eX(k)  when  xj&X(k),  zjGZi(k)  (4.7.2) 
Mij(Xi,Zi)feXi(k)  when  xiGXi(k),  z^Zj^k)  (4.7.3) 
Mij(xi»Zi)^X(k+1 )  when  xi&Xi(k),  zieZi(k)  (4.7.4) 
Fi(xi*zi)t:X(k+1 ).  when  xi£X(k+1),  z^Zj^k)  (4.7.5) 

Then,  if  all  initial  processor  estimates  x^O)  are  in  X(0), 
and  all  initial  measurements  z^o)  are  in  Zi(0),  the  limit 
points  of  (xi(t)}  are  solutions  for  each  i=1,...,N. 

We  interpret  the  theorem  as  follows.  Condition  (4.7) 
ensures  that  if  all  estimates  xi(t)  are  in  X^k)  and  all 
measurements  z^t)  are  in  Z^t),  then  eventually,  xA(t)  and 
zi(t)  will  enter  X^k+1)  and  Zi(k+1),  respectively,  and  remain 
there.  So  for  any  k>0,  x(t)£X(k)  for  large  enough  t.  Thus, 
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as  k  increases,  x(t)£X(k)  gets  arbitrarily  close  to  the 
solution  set.  To  see  how  (4.7)  guarantees  this,  consider  the 
following  argument. 

Let  us  assume  that  xi(t)£xi(k)  and  zi(t)^Zi(k)  for  all  i. 
Condition  (4.7.1)  says  that  after  processor  i  performs  a 
self-generated  update,  the  new  estimate  will  still  be  in 

Xi(k).  Similarly,  (4.7.2)  says  that  after  a  measurement 
update,  the  estimate  is  still  in  X^k). 

After  a  self-generated  update,  processor  i's  estimate  is 

in  xi(k).  We  may  regard  the  membership  of  xA(t)  in  X*(k)  as 
progress  toward  the  goal  that  xi(t)  eventually  be  in  Xi(k+1). 
Condition  (4.7.3)  says  that  that  progress  is  not  undone  by  any 
subsequent  measurement  updates. 

Condition  (4.7.4)  says  that  after  all  the  processors  have 
made  self-generated  updates,  and  enough  time  has  passed  for 
the  measurements  to  relfect  this,  then  processor  i's  next 
measurement  update  will  drive  xi(t)  into  Xi(k+1).  Finally, 
(4.7.5)  ensures  that  after  xi(t)  is  in  x^k+l),  additional 
self-generated  updates  will  not  push  x^t)  out  0f  Xi(k+1). 

Theorem  4.1  is  sufficiently  general  that  we  are  tempted 
to  try  to  reformulate  the  asynchronous  flow  control  algorithms 
in  terms  that  would  allow  us  to  apply  the  theorem.  The 
theorem  as  given,  however,  does  not  apply  to  the  algorithms 


described  in  Chapter  3  for  two  reasons, 


Theorem  4.1  states  that  if  an  asynchronous  algorithm 
meets  certain  conditions,  the  sequence  of  processor  estimates 
it  generates  must  converge  to  a  solution,  regardless  of  the 
manner  in  which  the  estimates  are  updated.  But  the  flow 
control  algorithms  in  Chapter  3  require  that  updates  only 
occur  at  specified  times,  that  is,  when  the  update  protocol  is 
satisfied. 


Furthermore,  Theorem  4.1  states  that  each  processor  i 
computes  a  complete  estimate  of  a  solution  x*,  whereas  the 
flow  control  algorithms  only  require  each  link  j  to  compute  an 
estimate  of  the  jth  coordinate  of  the  solution  p*. 


Both  of  these  difficulties  can  be  overcome  by  slight 
reformulations  of  the  theorem,  as  described  in  the  following 
sections. 


With  Partial 


In  this  section  we  show  how  to  modify  Bertsekas'  model  to 
describe  algorithms  in  which  each  processor  estimates  only  a 
partial  solution.  In  an  earlier  paper,  Bertsekas  [14]  gives  a 
result  similar  to  Theorem  4.1  for  algorithms  where  processors 
compute  only  partial  solutions,  but  that  result  is  less 
general  than  Theorem  4.1,  in  that  the  form  of  the  measurements 
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is  more  constrained. 


In  general,  the  dimensionality  of  the  solution  may  exceed 
the  number  of  processors.  In  that  case  we  would  need  some 
processors  to  calculate  estimates  of  several  solution 
coordinates.  However,  we  may  consider  the  ni  coordinates  that 
processor  i  estimates  as  an  vector,  and  then  consider  that 
vector  as  a  single  coordinate  of  an  N-vector.  So,  without 
loss  of  generality,  we  assume  that  each  processor  i  calculates 
only  the  ith  coordinate  of  the  solution.  Indeed,  we  may 
consider  that  in  the  general  algorithm  of  section  4.1,  the 
limit  points  that  each  processor  i  calculates  are  the 
(n-dimensional)  ith  coordinates  of  solutions  in 
(X«)NCxNoRnN. 

We  may  still  describe  the  algorithm  using  (4.1)  -  (4.3) 
by  simply  reinterpreting  x^(t)  as  processor  i’s  estimate  at  t 
of  the  ith  coordinate  of  the  solution.  We  write 

x(t)s( x-j (t ),..., Xfl(t) ) ,  where  x^(t)sXj  for  each  is1,...,N  and 
x(t)^XsXlx^#  .xXjj.  We  call  x(t)  the  complete  estimate  and 
xi(t)  the  ith  partial  estimate. 

We  might  now  state  a  theorem  similar  to  Theorem  4.1  for 
this  model,  but  for  one  remaining  difficulty.  Even  when  the 
processor  estimates  converge,  Theorem  4.1  does  not  promise 
that  the  different  processor  estimates  converge  to  the  same 
solution.  An  algorithm  where  the  processors  each  calculate 


one  coordinate  of  different  solutions  is  of  dubious  value. 
For  example,  we  might  have  x1  (t  )-^Xl  ««^X1  *  and  x2(t)-*x2«tX2«, 
but  (x  j  x2*)^-X*.  Additional  restrictions  are  needed  to 
guarantee  that  if  the  partial  estimates  converge,  they 
converge  to  coordinates  of  the  same  solution.  While  more 
general  results  may  be  possible,  we  choose  to  avoid  the 
problem  by  assuming  that  the  solution  set  X*  is  the  Cartesian 
product  of  sets  Xi*,  i=i,...,N, 

x*=x1*x...xXN«.  (4.8) 


Rather  than  restate  Theorem  4.1  for  this  model,  we  expand 
the  model  in  the  following  section  to  include  update 
protocols,  and  so  get  a  more  general  result. 

4.3  General  Asynchronous  Algorithms  with  Update  Protocols 

In  Chapter  3f  we  introduced  the  idea  of  an  update 
protocol  as  a  way  of  restricting  when  the  processors  could 
update  their  estimates.  In  this  section,  we  revise  our 
interpretation  of  an  update  protocol  so  that  the  processors 
may  update  at  any  time,  but  only  when  the  protocol  is 
satisfied  does  the  update  actually  affect  the  estimate. 

An  update  protocol  for  processor  i  can  be  expressed  in 
terms  of  a  protocol  function  ^xZ^  {0, 1 } ,  where  Zi  is  the 
set  of  all  possible  measurements  z^.  if  Pi(xi,zi)=1,  we  say 
that  the  measurements  zA  are  consistent  with  the  estimate  x< . 


Thus,  Pi(.)  is  the  indicator  fuction  of  some  subset  of  XiX£if 
and  we  call  this  set  the  consistent  set  Define 

Fi<xi*Zi)s (Fi(xi,ZjD  when  pi(xi»zi)=1 


when  Pi(xi,zi)=1 
otherwise 


(4.9) 


Mij(xi»Zi)sjMij(xi,zi)  when  Pi(xi,zi)s1 


otherwise. 


(4.10) 


We  call  Fi(.)  and  ^j(*)  the  constrained  update  functions,  and 
F^(.)  and  Mj^.)  the  unconstrained  update  functions.  We 
rewrite  (4.1)  and  (4.2)  as 

v  A  a 

i ( k+1 ) =Mi j (xi(t),zn  ( t ) ,  • . .  ,Zij(t), . . . , z im^( t))  (4.11) 

and 

xi(t+1 )sFi(xi(t),zi(t)).  (4.12) 

Hence,  we  restrain  the  processor  from  changing  its  estimate 
unless  the  processor's  current  measurements  are  consistent 
with  its  estimate. 


We  make  the  following  assumption  about  the  update 
protocol. 


Assumption  4.3.  For  each  processor  i,  there  exists  an 
infinite  sequence  of  times  for  which  Pt(xi(t),Zi(t))s1  and  the 
processor  updates  according  to  (4.12). 


One  way  to  ensure  that  Assumption  4.3  holds  is  to  require 


Pi(xi, (Gii (x1 ) » • • • *Gimi(x  i)))=1 


(4.13) 
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for  all  x...{xjL}... xX|j ,  1ij<mi,  for  all  x*  Xf. 

Essentially,  (4.13)  says  that  if  each  of  the  measurements 
could  have  been  generated  using  the  current  estimate  xif  then 
they  are  consistent  with  that  estimate. 


To  see  how  (4.13)  implies  Assumption  4.3,  suppose  there 
exists  some  time  tQ  such  that  Pj[( x^( t ) , zj,( t ) )=0  for  all  t>tQ. 
Then  xi(t)=xi(to)  for  all  t^tg.  But  by  Assumption  (4.2), 
there  exists  t^>tQ  such  that 

zij^t1 )  =  Gij(*i (Tij^(t))**.*,x^(tQ ),..., 

xN(TijN(t))).  (4.14) 

for  each  1<j<mi.  But  (4.13)  and  (4.14)  imply 

pi(xi(t1),zi(t)1)=1  (4.15) 

which  is  a  contradiction.  Hence  (4.13)  guarantees  the 

existence  of  an  infinite  sequence  of  times  for  which 
pi(xi(t),zi(t))s1 . 

We  give  the  following  theorem. 


Theorem  4.2.  Let  Assumptions  (4.0)  -  (4.3)  hold  and  let  X*  be 
of  the  form  (4.8).  Let  the  processor  estimates  xi(t)  be 
updated  according  to  (4.11)  and  (4.12).  Suppose  there  exist 
sequences  of  sets  {X^k)}  for  UiiN  with  the  following 
properties: 


and 


X*CX(k+1  )CX(k)C,.,a 
oo 

x*s  f)  X(k) , 

fc-i 


(4.16.1) 


(4.16.2) 
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where  X(k)=X1 (k)x. . .xXN(k) . 


For  1<i<N,  and  1^0,  define 

zij(k)={G1j(x) jx^X(k)}  (4.17.1) 

XiCkJrtFiCxi^i)  |xieXi(k),zie2i(k),Pi(xi,2i)s1}  (4.17.2) 
zij(k)  =  {Gij(x)|xt'x(k)),  (4.17.3) 

where  Zi(j£)  and  x(k)  are  defined  in  the  obvious  manner. 

Suppose  that  the  sets  X(k)  and  the  mappings  F^.),  Gj^.) 
and  M^.)  are  such  that,  for  all  i,  j,  k, 

Zi(k)£Xi(k)  (4.18.1) 

Mij(xi,zi)t-Xi(k)  when  x^Xj^k),  zf£rZi(k),  Pi(xi,zi)=1 

(4.18.2) 

Mij(xi, Zi)^Xi(k)  when  XjGXi(k),  ZifcZi(k),  Pj.(xi,zi)sl 

__  _  _  (4.18.3) 

Mij(xitzi)CXi(k+1 )  when  xitXi(k),  z^Zi(k),  Pi(xj,zi)s1 

_  (4.18.4) 

Fi(xi,ziKXi(k+1 )  when  xit-Xi(k+1),  zieZi(k),  Pi(xi,zi)=1 

(4.18.5) 

Then,  if  all  initial  processor  estimates  xi(o)  are  in  Xi(0), 
and  all  initial  measurements  zi(o)  are  in  Z^O),  the  limit 
points  of  (x(t)}  are  solutions. 

Proof.  We  will  show,  by  induction,  that  there  exists  a 

monotonic  increasing  sequence  of  times  (t^j  such  that 

xi(t)GXi(k)  for  UiiN  (4.19) 

for  all  t>tk.  Therefore,  by  (4.16)  and  (4.8),  the  limit 
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points  of  {x(t)J  are  solutions 


We  begin  by  showing  that 

xi(t)<£Xi(0)  for  liiiN  (4.20) 

for  all  t>tQ-o.  By  assumption,  (4.20)  holds  for  t=0.  Now 
suppose  there  exists  t?J>0  such  that  (4.20)  holds  for  all 
Oitit*.  Our  model  assumes  there  is  exactly  one  processor  i 
that  performs  an  update  according  to  either  (4.11)  or  (4.12) 
at  time  t'.  So  for  each  processor  j=i,  xi(t *+1 )=Xi(t' ) .  If 

Pi,( xi ( t * ) ,Zi(t' ))=0,  then  x^(t’+1 )  =  Xi(t* )  and  (4.20)  holds  for 
t'+1.  If  Pi(xi(t* ) , 2^(t 1 ) )=1 ,  we  consider  two  cases. 

Suppose  processor  i  receives  a  measurement  )  at  t1 

and  updates  according  to  (4.11).  By  the  causality  assumption, 

xhtTijh(t‘ ))exi(0)  for  each  UhiN.  Thus  (4.3)  and  (4.17.1) 
imply  that  z.j(t«  JeZ^O) .  Similarly,  )<^Zih(0)  for 

15hlrai.  Hence,  by  (4.18.2),  x^t ' +1 )feXi(0) . 

Now  suppose  processor  i  updates  according  to  (4.12)  at 
t».  As  above,  we  argue  that  zih(t •  )<£-Zih(0)  for  Uh^m*. 
Hence,  by  (4.17.2)  and  (4.18.1),  x^ t » +1  )67i( 0)cxi(0) . 

Thus,  (4.20)  is  satisfied  for  t'+1,  and  by  induction, 
(4.20)  holds  for  all  t20. 

Now  suppose  there  exists  k  and  tk  SUch  that  (4.19)  holds 
for  all  t>tk.  We  will  show  that  there  exists  tk+1>tk  such 
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that  (4.19)  holds  for  all  t>tk+^ .  We  do  this  by  constructing, 


for  each  i,  an 

intermediate 

sequence  of 

times, 

- 

tk<^ki<tki^tk'<rki* 

<tki,itk* 

1 ,  such 

that 

zi(t^Zi(k) 

for 

t>-rkl 

(4.21.1) 

xi(t)^:Xi(k) 

for 

t>tki 

(4.21.2) 

Xj  (t)6rXj(k) 

for 

t>tk', 

for  all  j 

(4.21.3) 

zi(t)^7i(k) 

for 

(4.21.4) 

xi(t)<£Xi(k+1) 

for 

fc>tki ' 

(4.21.5) 

xj(t)4Xj(k+1) 

for 

t>tk'«, 

for  all  j 

(4.21.6) 

m- 

By  the  induction 

hypothesis  and 

the  causality  assumption, 

there  exists  ‘c“'ki  for 

each  processor 

i  such  that 

(4.21.1) 

holds. 

We  claim  that  (4.21.2)  holds  when  tki  is  the  time  of 
processor  i‘s  first  update  after  ^"ki  according  to  (4.12)  with 
Pi(xi(t),zi(t))=1 .  Clearly  (4.21.2)  holds  for  tstki+l.  Now 
suppose  (4.21.2)  holds  for  some  t>tki+1 .  If  processor  i  does 

not  update  at  t,  or  Pi(xi(t),Zi(t))=0,  then  Xi( t+1 )  =  Xi( t)  and 
(4.21.2)  holds  for  t+1.  If  processor  i  updates  by  (4.11)  with 

Pi(xi(t),zi(t))  =  1,  then  by  (4.18.3),  xA( t+1  )eli(k) .  If 
processor  i  updates  by  (4.12)  with  Pi(xi(t),zi(t))s1 ,  then 
xi(t+1)6ri(k)  by  (4.17.2).  Hence,  (4.21.2)  holds  for  all 
t>tki. 


Now  choose  tk«-max  tki  and  (4.21.3)  is  satisfied. 

z 
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By  the  induction  hypothesis  and  the  causality  assumption, 
there  exists  ^ki»  for  each  processor  i  such  that  (4.21.4) 
holds. 

By  the  continuing  update  assumption  and  (4.18.4)  there 
exists  t^i  such  that  (4.21.5)  holds  for  tst^^'+l.  Now 
suppose  (4.21.5)  holds  for  some  t>tki«+i.  if  processor  i  does 
not  update  at  t,  or  Pi(xi(t),Zi(t))  =  0,  then  x^( t+1 )sxi( t )  and 

(4.21.5)  holds  for  t+1.  If  processor  i  updates  by  (4.11)  with 

pi(xi(t)tzi(t))=1,  then  by  (4.18.4),  x^ t+1 )£Xi(k+1 ) .  If 
processor  i  updates  by  (4.12)  with  pi(xi(t),Zi(t))=1 ,  then 

xi(t+1 )£Xi(k+1 )  by  (4.18.5).  Hence,  (4.21.5)  holds  for  all 
t>tki' • 

Hence,  we  may  choose  tk«  ’=max  tki*  and  t^st^1  '+1 .  We 
have  constructed  the  sequence  {tk}  as  desired.  This  completes 
the  proof  of  Theorem  4.2. 

Note  that  Theorem  4.1  can  be  considered  a  special  case  of 
Theorem  4.2,  by  using  the  protocol  functions  P^x^.z^sl  for 

Zj[6Zi,  and  by  letting  X^(k)  in  (4.16)  equal  X(k)  in 

(4.5) ,  for  each  i. 

While  we  could  use  Theorem  4.2  to  prove  the  results  of 
Chapter  3,  it  is  more  convenient  to  apply  the  following 
corollary.  The  corollary  is  just  a  simplified  form  of  Theorem 
4.2,  for  the  case  where  there  are  no  measurement  updates,  only 
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self-generated  updates 


1  ?ry  iLtl*  Let  the  processor  estimates  x^t)  be  updated 
according  to  (-4.11)  and  (it. 12)  with  Mij(xi,zi)=Xi.  Let 
Assumptions  4.0  -  4.3  hold  and  let  X*,  (X(k)},  (X(k)}, 

^Zij(k)}  and  {Z ^ j ( k ) }  be  as  in  the  statement  of  Theorem  4.2. 
Suppose 

”x(k)£x(k+1).  (4.22) 

Then,  if  xi(0)^xi(0)  and  zi(0)^Zi(0)  for  all  i,  the  limit 
points  of  (x(t)}  are  solutions. 

Proof.  We  prove  the  corollary  by  the  application  of  Theorem 

4.2.  Since  Mi(xi,zi)=Xi,  the  conditions  (4.18.2)  and  (4.18.3) 
are  trivially  satisfied.  The  condition  (4.22)  implies 

(4.18.4)  and  (4.18.1).  Furthermore,  (4.18.1)  implies 

Zij(k)={Gij(x) |x£X(k)} 

—  ^Gi j ( x ) I xeX(k) } 

=Zij(k)  (4.23) 

and  so 

tFi(xjL,Zi)  lxj£Xi(k+1 )  ,zi<&Zi(k),Pi(xi,zi)5l } 

-tFi(Xi,zi) ixieXi(k),zifeZi(k),Pi(xi,zi)s1} 
s**i(k) 

^xi(k+1).  (4.24) 

Hence  (4.18.5)  is  satisfied. 


We  have  shown  that  the  conditions  of  Theorem  4.2  are 
satisfied  and  therefore,  the  limit  points  of  (x(t)}  are 


solutions.  This  completes  the  proof  of  Corollary  4.1. 

In  the  next  section,  we  use  this  corollary  to  obtain  a 
result  that  shows  how  some  synchronous  algorithms  can  be  made 
to  work  asynchronously. 


4 . 4  Update 


for 


In  this  section  we  describe  how,  starting  with  a 
synchronous  distributed  algorithm  taken  from  a  given  class  of 
algorithms,  we  can  design  an  update  protocol  that  allows  us  to 
implement  the  algorithm  in  an  asynchronous  manner. 
Unfortunately,  it  is  not  always  possible  to  implement  the 
desired  protocol,  and  so,  the  result  has  limited  application. 
We  present  it  mainly  for  the  insight  it  provides  about  why 
such  update  protocols  work. 

We  model  a  synchronous  algorithm  as  follows.  As  in 
section  4.2,  each  processor  i  at  time  t  has  an  estimate 
of  the  ith  coordinate  of  a  solution  x*€X*t  where  X* 
is  of  the  form  (4.8).  All  the  processors  update  their 
estimate  simultaneously  at  each  t,  according  to 

*1(t+1)sFi(xi(t),zi(t))  (4.25) 


where 


Note  that  there  are  no  measurement  updates,  since  all 
measurements  are  "received"  simultaneously.  Combining  (4.25) 
and  (4.26)  we  get 

x(t+1)=(H1(x(t))|...|HN(x(t))) 

=H( x( t ) )  (4.27) 

where  Hi(x)_Fi(Xi,Gi(x) ). 

We  give  the  following  theorem. 

Theorem  4 .3 .  For  each  i,  let  (X(k)}  be  a  sequence  such  that 
X*a(k+1)CX(k)C...Cx  (4.28.1) 

Qo 

X«=  f\  X(k)  (4.28.3) 

k=-| 

and 

H(X(k))CX(k+1),  (4.28.3) 

where  X(k)sX1  (ic)x. .  .xXN(k)  and  H(X(k))  is  the  image  of  X(k) 
under  the  mapping  H(*). 

Let  tx(t)}  be  generated  by  (4.3)  and  (4.9)  -  (4.12)  with 
x(0)GX(0)  and  z ^(Ote-Zj^O)  for  each  i.  Let  Assumptions  4.1  and 
4.2  hold.  If  it  is  possible  to  define  Pi(xi,zi)  such  that 
tFi<xi»Zi) lxi^Xi(k),ZieZi(k)fPi(xi,zi)s1 } 

il<Fi(Xi,Gi(x)))lx  X(k) } ,  (4.29) 

and  such  that  Assumption  4.3  holds,  then  the  limit  points  of 

(x(t))  are  solution*. 

Proof.  We  prove  Theorem  4.3  by  the  application  of  Corollary 


4.1.  Since  there  are  no  measurement  updates,  we  take 
Mij(*i»zi)s*i*  Combining  (4.17.1),  (4.17.2)  and  (4.29)  we  get 

Xi(k)={Fi(xi,zi)  |xj6-Xi(k) ,zi<SZi(k;,Pi(xi,zi)s1 } 

-{Fi(*i,Gi(x))  ix^X(k)} 

s<Hi(x)|xeX(k)} 

sHi(X(k)) 

— Xi(k+1 ) .  (4.30) 

Hence,  (4.22)  holds  and  the  conditions  of  Corollary  4.2  are 

satisfied.  This  completes  the  proof  Theorem  4.3. 

The  first  part  of  the  theorem  simply  states  conditions  on 
the  manner  of  convergence  of  the  estimates  generated  by  the 
synchronous  algorithm.  Clearly,  if  x(0)6X(0),  the  sequence 
(x(t)}  generated  by  the  synchronous  algorithm  (4.27)  is  such 
that  x(k)£X(k)  for  all  k,  and  hence  its  limit  points  are 
solutions. 

Now  consider  an  '  asynchronous  algorithm  that  satisfies 
Corollary  4.1.  The  estimate  sequence  (xi(t)}  behaves  as 
desired  because  (4.22)  guarantees  that,  with  x^(t)€X^(k)  and 

Zi(t)€Zi(k),  updating  will  never  cause  xA(t)  to  go  back  to 
Xj^k-I),  and  because  Assumption  4.3  guarantees  that  x^(t)  will 
eventually  enter  Xi(jc+1).  Now  take  an  algorithm  for  which 
(4.22)  does  not  hold.  If  we  choose  a  protocol  such  that 
updating  is  forbidden  whenever  updating  would  cause  x^(t)  to 
go  back  to  XjCk-i),  then  (4.22)  will  be  satisfied. 


Equation  (4.29)  tells  us  how  to  design  that  update 
protocol.  We  do  this  by  comparing  the  image  of  Xi(if)xzi(k) 
under  F^.)  with  the  image  of  XjL(k)XGi(X(k) )  under  F^*).  The 
set  Xi(k)XQi(x(k) )  consists  of  all  estimate  and  measurement 
pairs  for  processor  i  such  that  each  of  the  measurements  could 
have  been  generated  from  the  same  element  x€X(k).  Hence, 

xi(k)xGi(X(k) )  is  a  subset  of  Xi(k)xZ^(k).  Now  choose  the 
consistent  set  such  that  F^(*)  maps  Xi(k)xzi(k)^Ci  to  the 
image  of  X^iOxGiCXCk))  under  Fi(*)*  Then,  with  the  protocol 
function  equal  to  the  indicator  function  on  (4.29)  holds. 
The  idea  behind  (4.29)  is  that  updates  are  permitted  only  when 
updating  will  not  push  xjL(t)€Xi(k)  back  into  Xi(k-I). 

While  we  can  always  select  the  consistent  set  so  that 
(4.29)  holds  for  any  given  k,  it  may  not  be  possible  to  choose 
the  set  so  that  (4.29)  holds  for  all  k.  Even  if  (4.29)  holds 
for  all  k,  the  update  protocol  may  not  satisfy  Assumption  4.3. 
Furthermore,  it  may  not  be  possible  to  implement  the  resulting 
update  protocol,  since  doing  so  might  require  the  processors 
to  know  the  exact  form  of  the  measurement  generation  functions 

Gij<*)- 

Though  these  limitations  restrict  the  usefulness  of 
Theorem  4.3»  it  still  provides  a  starting  point  for  someone 
trying  to  design  an  update  protocol. 


4.5  lh&  Flow  Control  Algorithm  as  an  Example 


In  this  section,  we  give  an  alternate  proof  of  Theorem 


3.2,  using  Corollary  4.1.  In  fact,  we  prove  a  somewhat 
stronger  result  than  Theorem  3.2,  since  we  will  show  that  the 
update  protocol  is  not  needed  for  the  controls  to  converge. 
We  note,  however,  than  in  order  to  apply  Corollary  4.1,  we 
must  make  the  causality  assumption,  which  was  not  required  for 
the  proof  of  Theorem  3.2.  The  update  protocol  is  required, 
even  with  causality,  for  the  algorithm  of  Theorem  3.3,  as  a 
simple  example  will  show. 


We  begin  by  showing  how  the  elements  of  the  flow  control 
model  fit  the  general  model  of  sections  4.2  and  4.3.  The 
feasible  set  X  is  taken  to  be  RN,  where  N  is  the  number  of 
links  in  the  network,  and  the  solution  set  is  X*={p*},  where 
p*  is  the  fair  allocation  over  Hayden's  feasible  set  (3.27.1). 
The  processors  are  the  links  and  the  estimates  they  compute 
are  the  coordinates  of  the  control  vector  p.  The  measurements 
zji  received  by  link  j  are  just  the  rates  r^  of  its  sessions. 
Since  the  links  receive  new  measurements  continuously,  but  do 
not  update  each  time  new  measurements  are  received,  the  only 
events  of  interest  are  the  control  updates  which  take  place  at 
times  tjn,  for  j t'^,  n20.  In  the  notation  of  Chapter  4,  we 
have 


and 


Mij(Pj»rj)=Pj 
Fj(Pj*rj>smax  r 

d 


ij+(cj-^-  rij)/wj 

3 


(4.31.1) 

(4.31.2) 


(4.31.3) 


\ 


cij(p)s  min  Pfc  for 

The  original  protocol  function  is  given  by 
Pj(Pj.rj)  =  fl  if  max  fijiPj 

I  ***t 

[0  otherwise.  (4.32) 

However,  we  will  show  that  the  conditions  of  Corollary  4.1  can 
be  met  using  *j(pj,rj)=1  for  all  Pj.r/’R. 

To  apply  the  corollary,  we  must  construct 
^Xj(n)}  such  that 

Xj(n+1)  £Xj(n)£  ...  ^Xj(O). 

tX> 

^  xj(n)={pj«} 

and 

FJ<Pj.rj*Xj(n.1> 
when  Pj£Xj(n)f  rj£Zj(n). 

Even  though  we  have  shown  in  Chapter  3  that  the  controls 
of  the  flow  control  algorithm  converge  for  any  initial 

controls  Pj(o)£R  and  rates  rij(0)£R,  there  exist  networks  for 
which  it  is  not  possible  to  construct  a  chain  of  sets  X(n)  as 
in  (4.33)  with  X(0)sRN.  This  is  because,  with  X(0)sRN, 
X(0)sRn  which  implies  X(1)sRN.  So,  by  induction  X(n)*RN  for 
all  n.  Instead,  for  any  initial  Pj(0)  and  rij(O),  we  take 

Xj(0)S[-Aj|Bj], 
for  suitably  large  Aj(  Bj 


the  sequences 

(4.33.1) 

(4.33.2) 

(4.33.3) 


_  ion  _ 


(4.34) 


Now  partition  the  links  into  sets  A  and  the 

*  ^ 

sessions  *1  into  sets  A- 1 , . . . » Jll  as  in  the  proof  of  Theorem 
3.2.  We  will  construct  {X(n)}  by  finding  sequences  (ek(n)} 
and  (Ek(n)}  for  each  k=1,...,L,  such  that 

Xj(n)  =  tpk-ek(n),pk+Ek(n)]  for  (4.35) 

and  ( X j ( n ) }  satisfies  (4.33). 

Before  defining  (ek(n)}  and  (Ek(n)},  we  describe  the  idea 
behind  their  construction.  We  begin  by  taking  e"*(n)  =  0  for  n21 
and  finding  (E^n)}  that  is  monotonically  decreasing  and 
converges  to  0.  As  long  as  E^(n)  is  greater  than  some 
threshold  less  than  p2,  we  keep  Ek(n)  fixed  at  some  large 
number  E  and  let  pk-ek(n)  =  p1-e1  (n) ,  for  all  k22.  When  E'Un) 
drops  below  the  threshold  at  some  time  N2j  we  iet  p2-e2(n) 
rise  above  p1  and  E2(n)  begins  to  fall.  We  still  keep  Ek(n) 
fixed  at  E  and  let  pk-ek(n)=p2-e2(n)  for  k^3.  This  process  is 
repeated  until  (ek(n)}  and  (Ek(n)}  have  been  found  for  all  k. 


We  formally  construct  the  sequences  (ek(n)),  { Ek ( n ) }  as 


follows.  First  define,  for  ks1t...,L, 
ek(0)s  aE 


(4.36.1 ) 


Ek(0)s  E 


(4.36.2) 


where  a=  min  1/Wj  and  g  iS  some  suitably  large  constant  such 
that  E2pL/a.  Now  let 

e1(n)=0  (4.37.1) 


[•] 


E' (n+1 )=( l-a)E 


(4.37.2) 


for  all  n21.  Suppose  that  we  are  given  a  sequence 


**1<N2<. .  .<Nl  such  that  N-jal  and,  for  k>1, 
ek”1  (nJ+E^"1  (n)<(pk-pk“1  )a/(  1-a) 

for  all  n>Nk>  Now  define,  for  n>Nk 
ek(n+1)=(1-a)(ek(n)+Ek-1(n)) 

Ek(n+1 )=( 1-a) (ek(n)+Ek(n) ) 
and  for  N^n<Nk+lf  K>k, 

eK(n+1)s  pK_pk+ek(n+1 ) 

EK(n+1)=E. 


(4.38) 


(4.39.1) 

(4.39.2) 


(4.40.1) 

(4.40.2) 


In  Appendix  0,  we  show  by  induction  that  the  sequence 

N1 <N2<. . .<Nl  satisfying  (4.38)  exists,  and  that  the  sequences 
{e*(n)}  and  (Ek(n)}  are  monotonically  non-increasing  and 
converge  to  0,  for  all  k.  Hence,  (4.33.1)  and  (4.33.2)  are 
satisfied. 


Furthermore,  we  show  that  the  sequences  have  the 


properties  that  for  k<K,  for  all  n, 
pk-ek(  n  XpK-eK(  n ) 
elc(n)ieK(n) 


(4.41.1) 

(4.41.2) 


E*(n)iEK(n). 


(4.41.3) 


These  properties  will  be  needed  to  show  that  (4.33.3)  holds. 


We  show  that  (4.33.3)  is  satisfied  for  nsO.  Note  that 
<Wj-1)(pk-aX)+  ®ax  riJi  £  rij  max  r^,  (4.42) 


jfcZj(O).  Therefore, 

Fj(Pj»rj)>  nax  r ij+(c j-Wj  max  rij)/Wj 

.  *  } 

^Vwj 

2P1 


(4.43) 


Also, 


rj(Pj.rj)i  max  rij+(CJ-(Wj-1 ) (pk“aX>-  max  r^j/Wj 

<(1-1/Wj)(  max  rij-pk+aX)+Cj/v/j 

A(:\ 

<(1-1/WJ)(E+aE)+Cj/Wj 

<(1-a)(1+a)E+pk 


<p*+E. 


Hence, 


Fj(Pj,rj)e-[p1»pk+E3=xj(i), 


(4.44) 


(4.i*5) 


as  desired. 


Before  showing  that  (4.33)  holds  for  n21,  we  make  the 
following  observations.  If  session  i  on  link  j  is  in^k, 
then  every  link  in  its  path  is  in  for  some  K2k.  Thus, 

(4.31.3)  and  (4.41.1)  imply 
min  pK-eK(n) 


=pk_ek(n) 


for  i^2-k»  rij^Zij(n).  Furthermore, 

rij4pk+Ek(n) 

for  i^ik,  rij^Zij(n). 


(4.46) 


(4.47) 


Now  we  show  that  (4.33)  holds  for  n21 .  First  we  show 
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that 


Fj (pj ,  i- j  )  (4.48) 

for  j^.»CK»  Pj£Xj(n),  rj^-Zj(n).  We  consider  two  cases:  n<NK 
and  n>NK.  if  n<NK,  by  (4.44) 

Fj ( p j , r j ) <pK+E= pK+EK(n+i ) .  (4.49) 

Now  suppose  n>NK#  Let  Wjk  be  the  number  of  sessions  on  link  j 
in  Xk-  Then,  by  (4.46) 

Wjk(pk-®k(n) )+max  rt ,_(pk-ek(n) ) 

^  i 

sCj_^_  Wjkek^n)  +raax  ri^(pk_ek(n) ).  (4.50) 

Therefore, 

K- 

Fj(Pj.rj)<max  rij+  [S  Wjkek(n)-  max  rij+pK-eK(n) ]/Wj 

Jz-y 

=  <  1“1/wj)max  rij+t^  Wjkek^n^+PK“eK^n^/Wj 

<(1-1/Wj)(pK+EK(n))+eK(n)+[pK-eK(n)]/Wj 
=PK+( 1-1/Wj) (gK(n) )+eK(n) ) 
iPK+(1-a)(EK(n)+eK(n)) 

=pK+EK(n+1),  (4.50) 

as  desired.  The  third  step  follows  from  (4.47)  and  (4.41.2). 


Next  we  show  that 

Fj (pj , rj )->p^-eK(n+1 )  (4.52) 

for  j€XK»  Pj6Xj(n),  rj^Zj(n). 


First  we  derive  some  preliminary 
convenience,  define 


results.  For 

(4.53.1) 
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and 


ajKsWjK/Vj, 


(4.53*2) 

Then  WjK  is  the  number  of  sessions  on  link  j  that  are 
controlled  by  links  at  a  higher  level  than  K,  that  is,  the 


number  of  sessions  that  are  in 


Let  J  be  the  largest  number  for  which  that 
For  any  H^J,  (4.47)  implies 


H-l 


Xe-v- 


rii<  2.  W-jk(Pk+El<{n)  )+W,H-1max  riJ# 


(4.54) 


■h=\ 


Therefore 


F-j(pjfrj)>max  rij+[cj-5_  Wjk(Pk+EkCn)  >-WjH-1na«  rij]/Wj 


fesj 


/.ex. 


>(  1-a 4H-1  )max  r„ 


*  4  '  /M*BA  I  .i  1 

J  ‘  ■LJ 


< 


H-I  _  fc' 

+a  .H-1  (C  <r  W ,kpk)/w.H-1.  Z  a ^Ek.  (4.55) 
J  J  K,-#  J  J  /c®  |  J 

By  the  construction  of  the  fair  control  vector 


H-l 


PH<(c  ..  2  WikPk)/WiH-1  • 
J  Jpi  J  J 

By  (4.41.3) 


(4.56) 


H-l  H-l 

2*  a  kEk(n)<  2  a^En-1  (n) 
—  J  K=l  J 


K=| 


=(1-ajH-l )EH-1(n), 


(4.57) 


and  by  the  choice  of  J  and  (4.41.1) 


max  r1j^pJ-eJ(n) 


2pH_eH(n) . 


(4.58) 


Hence, 


Fj^pj*  )  (pH-eH(n)-EH“1  (n )  )+a jH-1  pH 


[I] 


=  pH-(1-a.H-1)(eH+EH-'l(n)) 

J 

2pH-(1-a)(eH(n)+EH-1(n)).  (4.59) 

We  are  ready  to  show  that  (4.52)  holds.  We  consider  two 
cases:  JsK  and  J<K.  Suppose  J=K.  If  n>NK>  then  (4.59) 
implies 

trj(pj,rj)ipK-(1-a)(eK(n)+EK“1  (n)) 

=pK-eK(n+1).  (4.60) 

If  n<NK>  let  H  be  the  iargest  number  such  that  n>NB.  Then 
Fj(Pj»rj)->PH"eH(n+1)» 

spK_eK(n+i ) .  (4.61) 


Now  suppose  J<K.  Then  (4.61)  also  holds  for  n<NJ+i. 
Finally,  suppose  n>NJ+1>  By  (4.47) 

cr 

^.Wjk(Pk+Ek(n))+®ax  rij.(pJ+EJ(n)).  (4.62) 

Thus, 

Fj<Pkj»rj)2(1-1/Wj)niax  rij 

*  T" 

+(c.  >  Wik(pk+Ek(n))-pJ-EJ(n))/Wi 
J  A=\  J  J 

>(1-1/Wj)(njax  rij-E*^^n) )“PJ<,Wj+(cj-21  WjkPk)/Wj 

1 

>(  1-1/W.) (pJ-eJ(n)-E«)(n) )-pJ/Wj,‘,(cj-^£  Wjkpk^Wj 
~T 


>PJ+(c..^  WikPk)/Wi-(i-a)(eJ(n)4-EJ(n)) 
J  k=\  J  J 

=  pK-(1-a)(eJ(n)+E‘I(n)), 


(4.63) 
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where  the  last  step  follows  from  the  construction  of  the  fair 
control  vector.  But,  since  n>NJ+lf 

(1-a)(eJ(n)+EJ(n))^(1-a)(eJ+Un)+EJ(n)) 

=  e«J+1  (n+1 ) 

ieK(n+1)  (4.64) 

and  so 

Fj<Pj.rj)->pK-«Kt"*'>.  (*-65) 

We  have  shown  that, 

Fj(Pj»r j)6[pK-eK(n+1 ) ,pK+EK(n+1 ) ] 

=Xj(n+1)  (4.66) 

for  Pj6rXj(n),  rj€^j(n).  The  conditions  of  Corollary 

4.1  are  satisfied,  and  therefore,  the  controls  generated  by 
the  asynchronous  flow  control  algorithm  of  Theorem  3.2 
converge  to  p*  without  the  update  protocol. 

It  is  instructive  to  consider  why  the  proof  in  Chapter  3 
requires  the  update  protocol  while  this  proof  does  not.  In 
both  cases  we  must  show  that,  for  any  e>0,  the  control  p.(t) 

J 

eventually  drops  below  PjS+e.  a  necessary  step  in  showing 
this  is  demonstrating  that 

Pjn+1iPjn”m+(cj-f jn)/wj  (4.67) 

for  some  non-negative  m.  With  the  causality  assumption, 

max  rij(tjn)iPjn"m  (4.68) 

for  some  m^O,  and  so  (4.67)  is  satisfied.  With  the  update 
protocol,  (4.68)  holds  for  msO  and  the  causality  assumption  is 


107  - 


not  needed 


This  leads  us  to  question  whether  the  update  protocol  is 
required  for  the  generalized  algorithm,  as  given  in  Theorem 
3.3,  if  we  assume  causality.  In  fact,  the  protocol  is  needed, 
as  the  following  example  shows. 

Consider  a  network  of  one  link  with  capacity  c  and  W 
sessions.  Let  g(x)=x  and  bi(x)  =  x>  for  an  i.  Then 

p(n+1 )s(1-a)p(n)+a(c-f(n) ) ,  (4.69) 

where  a=1/(W+1 ) .  Note  that  p*sac.  Now  suppose  that  f(0)sc, 
and  the  link  updates  many  times  in  rapid  succession,  so  that 
f(0)  does  not  have  time  to  change  from  one  update  to  the  next. 
Then,  after  N1  updates, 

P(N1)=(1.a)Nlp(0)+(1-aNl)Cc-f(0)) 

sC-f(0)-e(N1)>  (4.70) 

We  can  make  e(N.|)  as  anaii  as  we  like  by  taking  large  enough 

N1.  Now  the  link  waits  until  the  flow  reflects  this  control, 
that  is, 

f^l)_W(c.f(0)_e(N1)).  (4.71) 

Once  again,  the  link  updates  rapidly,  without  waiting  for  the 
flow  to  change,  so  that 
pCN2)-c-f(N1)-e(N2) 

=c-W(c-f(0)-e(N1))_e(N2).  (4.72) 

In  this  manner,  we  can  construct  a  sequence  of  times  {Nk}  such 
that  the  flows  are  given  by 


f<Nk+1)sW(c-W(c-f(Nk)-e(Nk))-eCNk+1)).  (4.73) 

For  appropriately  chosen  (e(Nk)}f  this  sequence  of  flows  will 
never  converge. 

Let  us  see  where  the  conditions  of  Corollary  4.1  fail  to 
hold  for  this  example.  Suppose  we  have 

X(n)s[p*-e(n) , p*+E(n) 3 .  Then  Zt(n).X(n)  for  each  i.  Now 
suppose  that  p(n)=pB-e(n)  and  ri(n)=p»+E(n) .  Then 
p(n+1 )s( 1-a)(p*-e(n) )+a(c-W( p»+E(n) ) 

sp*-( 1-a)(e(n)+E(n)).  (4.74) 

Hence,  we  cannot  guarantee  that  F(p,r)£X(n)  when  p€X(n), 
r  Z(n).  Thus,  for  this  algorithm,  the  update  protocol  is 
required  to  ensure  that  the  controls  get  above  a  given 
threshold.  Examining  the  proof  of  Theorem  3.3,  we  see  that 
the  update  protocol  was  invoked  for  that  very  reason. 

We  conclude  this  chapter  by  remarking  that  these  results 
are  not  obvious.  Without  applying  Theorem  4.2  to  the  flow 
control  algorithms,  we  would  probably  have  never  discovered 
that  one  algorithm  requires  the  update  protocol  and  the  other 


does  not 


Chapter  5 


Simulation  and  Results 

In  this  chapter  we  describe  a  computer  program  designed 
to  simulate  a  voice  packet  network  using  a  flow  control 
algorithm,  such  as  one  of  those  described  in  Chapter  3.  The 
program  was  written  in  Lisp  on  the  Symbolics  3600,  by  Allan 
Wechsler  and  myself.  The  program  listing  is  given  in  Appendix 
E.  After  describing  the  program  model  and  the  program,  we 
introduce  a  specific  network  model  and  discuss  the  results  of 
the  simulation  using  that  model. 

5.1  Iii£  Simulation  Model 

The  simulation  model  is  substantially  the  same  as  that 
used  by  Hayden  [9],  with  some  minor  differences. 

The  program  allows  the  user  to  define  a  network  with  an 
arbitrary  topology.  The  network  is  specified  by  two  global 
variables:  a  list  of  its  users  (sessions)  and  a  list  of  its 
links. 


A  user  is  a  data  object  that  has,  among  other  attributes, 
a  rate,  a  partner  (the  other  user  that  it  talks  to)  and  a 
route  (the  list  of  the  links  that  are  in  its  path).  If  user 


i's  partner  is  user  k,  then  user  k's  partner  is  user  i.  At 
any  given  time  a  user  is  either  active  (talking)  and  its 
partner  is  inactive  (silent),  or  vice  versa. 

When  a  user  is  active,  it  generates  a  variable  length 
voice  packet  approximately  every  20  msec.  The  actual  time 
between  packet  generations  is  a  random  variable  uniformly 
distributed  between  18  and  22  msec.  The  length  of  a  voice 
packet  in  bits  is  given  by 

PACKET-LENGTHs  [uSER-RATE/5ol .  (5.1 ) 
The  rate  is  a  floating  point  number,  but  the  length  is  an 
integer. 

After  generating  a  voice  packet,  an  active  user  will 
generate  another  voice  packet  with  probability  (1-p). 
Otherwise,  the  packet  generated  is  the  last  in  its  talk-spurt. 
If  the  packet  is  designated  last,  the  user  becomes  inactive 
after  the  packet  is  transmitted  and  the  partner  becomes  active 
when  the  packet  is  received.  We  use  psl/60,  and  hence,  the 
number  of  packets  per  talk-spurt  is  a  geometric  random 
variable  with  mean  60.  This  conforms  with  experimental  values 
measured  by  Brady  [15],  who  found  actual  talk-spurt  durations 
to  be  approximately  exponentially  distributed,  with  mean  1.2 
sec.  This  is  not  a  completely  accurate  representation  of  real 
speech,  however,  since  brief  periods  of  silence  usually  occur 
between  talk-spurts,  and  talk-spurts  do  not  always  alternate 
strictly  between  two  members  of  a  conversation.  Nevertheless, 


it  is  an  acceptable  approximation  for  our  purposes.  We  also 
have  the  option  of  setting  the  talk-spurt  length  to  infinity. 
This  lets  us  study  the  steady  state  behavior  of  the  system. 

When  a  user  is  inactive,  it  generates  fixed  length 
control  packets  at  regular  intervals,  for  the  purpose  of 
passing  feedback  information  to  its  partner.  The  length  of 
the  control  packets  is  10  bits,  and  the  time  between  the 
generation  of  successive  control  packets  is  100  msec. 

A  link  is  a  data  object  that  has  a  control  value,  a  flow, 
a  list  of  its  currently  active  users  and  a  queue  of  packets 
waiting  to  be  transmitted  on  the  link. 

A  link's  list  of  its  active  users  also  includes,  for  each 
user,  the  user's  rate,  as  determined  by  examining  the  most 
recently  received  packet  from  that  user.  This  rate  list  is 
used  to  determine  whether  the  update  protocol  is  satisfied, 
and  to  calculate  the  flow  of  the  link.  This  is  the  main 
difference  between  our  simulation  and  Hayden's.  In  Hayden's 
program,  a  link  determines  its  flow  by  observing  the  number  of 
bits  arriving  in  the  queue  over  a  given  period  of  time.  This 
affects  the  value  of  the  flow  in  three  ways  not  accounted  for 
by  the  theoretical  model.  First,  the  observed  flow  includes 
control  packet  traffic.  Also,  the  observed  flow  for  a  link  is 
limited  by  the  capacity  of  the  neighboring  links  that  feed  its 
queue.  Finally,  the  observed  flow  may  represent  the  rates  of 


the  link's  different  users  in  unequal  proportions.  For  these 
reasons,  we  prefer  to  calculate  a  link's  flow  by  summing  the 
rates  of  the  link's  active  users. 


I 

I 

I 

I 


Each  link  attempts  to  update  its  control  periodically,  by 
first  checking  whether  the  specified  update  protocol  is 
satisfied.  If  it  is,  the  link  updates  according  to  the 
specified  update  function.  If  not,  the  link  waits  a  given 
interval  and  tries  again.  There  are  two  update  interval 
parameters  that  can  be  adjusted:  the  time  between  a  succesful 
update  and  the  next  update  attempt,  and  the  time  between  an 
unsuccessful  attempt  and  the  next  attempt.  They  are  called 
the  UPDATE-INTERVAL  and  the  UPDATE-ATTEMPT-INTERVAL.  In 
section  5.4  we  will  see  that  the  choice  of  these  parameters  is 
critical  to  the  performance  of  the  system. 

One  of  two  update  protocols  can  be  selected:  MOSELY  or 
HAYDEN.  The  MOSELY  protocol  is  given  by  (3.21)  and  the  HAYDEN 
protocol  always  permits  updates. 


Three  different  update  functions  can  be  selected.  The 
HAYDEN-UPDATE-FUNCTION  is  given  by 

Pj(t+1  )smax[Cj/Wj,min[c j, pj(t)  +  Ccj-f j(t ) )/Wj] ] .  (5.2.1 ) 

The  MOSELY- UPDATE- FUNCTION  is  given  by 
Pj(t+1 )=max[cj/Wj, 

mintCj ,max  rij(t)+(cj-f j(t))/Wj]].  (5.2.2) 
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The  JAFFE-UPDATE-FUNCTION  is  given  by 

Pj(t+1 )smax[cj/(Wj+1 ), 

minCcj.Pj(t)**-(cj-fj(t)-pj(t))/(Wj^1)]].  (5.2.3) 

The  variable  Cj  in  these  equations  represents  the  effective 
link  capacity  and  not  the  true  capacity.  For  the  simulation 
we  used  Cj-.8Cj,  where  Cj  is  the  actual  capacity.  These 
update  functions  differ  slightly  from  those  given  earlier,  in 
that  we  restrict  the  range  of  the  controls.  Since  we  know 
that  the  fair  control  allocation  can  never  result  in  a  control 
for  link  j  outside  the  interval  tcj/Wj,cj]  for  Hayden's 
feasible  set  or  tcj/(WJ+1 ) ,cj ]  for  Jaffe's  feasible  set,  these 
are  reasonable  modifications. 

A  packet  is  an  object,  created  by  a  user,  that  has  a 
source  and  destination  (two  users),  a  route  (a  list  of  links), 
forward  control  and  feedback  information,  and  a  variety  of 
statistics,  such  as  its  length,  time  of  generation,  and  the 
rate  of  its  source. 

When  a  packet  is  created,  its  forward  control  is  set  to 
infinity.  Each  time  a  packet  is  transmitted  across  a  link, 
its  forward  control  is  reset  to  the  minimum  of  its  current 
forward  control  and  the  control  of  the  link.  Hence,  when  a 
packet  arrives  at  its  destination,  its  forward  control  is 
equal  to  the  minimum  control  of  the  links  in  its  path.  This 
number  is  stored  by  the  destination  and  used  as  the  feedback 


value  In  the  next  packet  created  by  the  destination  to  be 
returned  to  the  source.  When  the  source  receives  the  returned 
packet,  it  changes  its  rate  to  equal  the  feedback  value. 


Note  that  a  packet  contains  its  source's  rate.  This  is 
desirable  for  two  reasons.  A  link  could  calculate  a  given 
user's  rate  by  multiplying  the  length  of  one  of  that  user's 
packets  by  50,  but  this  produces  serious  round-off  errors. 
Also,  a  user  might  sometimes  want  to  transmit  at  a  rate  lower 
than  that  assigned,  while  still  reserving  for  itself  the 
option  to  send  at  the  higher  rate  later.  Hence,  we  prefer  the 
users  to  communicate  their  rates  to  the  links,  rather  than  let 
the  links  measure  the  rates. 

This  completes  the  description  of  the  simulation  model. 

5.2  XbL£  filmulailan  Program 

In  this  section  we  describe  the  event  driven  program  that 
was  written  to  simulate  the  model  of  section  5.1. 

The  program  works  by  scheduling  and  performing  events  in 
an  event  table.  An  EVENT  consists  of  a  TIME,  a  FUNCTION  and 
ARGUMENTS.  When  an  event  is  created,  it  is  added  to  the 
•EVENT-TABLE*,  which  is  implemented  as  a  heap.  The  heap  is 
sorted  so  that  the  event  at  the  top  of  the  heap  is  always  the 
one  whose  TIME  is  earliest.  An  event  "takes  place"  when  it  is 


removed  from  the  top  of  the  heap  and  its  FUNCTION  is  applied 
to  its  ARGUMENTS.  When  this  happens  the  global  variable 
•TIME*  is  set  to  the  TIME  of  the  EVENT  being  performed. 

The  program  begins  by  intializing  the  network,  creating 
the  start-up  event  and  adding  it  to  the  heap.  The  program 
then  enters  a  loop  which  repeatedly  removes  events  from  the 
top  of  the  heap  and  performs  them,  until  the  global  variable 
"TIME*  exceeds  the  given  time  limit.  Since  most  events,  when 
performed,  create  one  or  more  new  events  with  times  later  than 
•TIME*,  the  heap  never  becomes  empty. 

There  are  ten  different  types  of  events  that  occur,  as 
described  below. 

SIllULAIIQN-SIARTUP 

This  is  the  first  event  performed.  It  schedules  the 
first  VOICE- PACKET-GENERATIONs  for  all  intially  active  users 
and  the  first  CONTROL- PACKET-GENERATIONs  for  all  initially 
silent  users.  While  the  first  group  of 
VOICE-PACKET-GENERATIONs  are  synchronized,  subsequent 
VOICE-PACKET-GENERATIONs  will  rapidly  fall  out  of 
synchronization.  The  same  is  true  of  the 
CONTROL-PACKET-GENERATION s.  This  event  also  schedules  the 
first  UPDATES  and  LINK-STATISTICS-COLLECTIONs  for  each  link, 
and  the  first  USER-STATISTICS-COLLECTIONs  for  each  user.  The 


times  of  the  first  link  UPDATES  are  randomized,  since  these 


events  would  not  fall  out  of  synchronization  otherwise. 


When  a  voice  packet  is  generated,  its  FORWARD-CONTROL, 
FEEDBACK-CONTROL  and  LENGTH  are  set,  as  described  in  the 
previous  section.  The  route  of  the  packet  is  set  to  the  route 
of  its  source,  its  GENERATION-TIME  is  set  to  *TIME»,  and  TYPE 
is  set  to  VOICE.  The  LAST-IN-TALK-SPURT?  flag  is  set  to  T  or 
NIL,  according  to  the  outcome  of  a  random  "coin  toss".  The 
entire  packet  is  scheduled  to  arrive  at  the  first  link  in  its 
route  at  «TIME«  +  •PACKET-GENERATION-DELAY*.  This  is 
accomplished  by  adding  the  events  PACKET-ARRIVAL  and 


PACKET-TAIL- ARRIVAL 


event-table. 


LAST-IN-TALK-SPURT?  is  NIL,  another  VOICE-PACKET-GENERATION 
is  scheduled  for  the  appropriate  future  time,  otherwise,  a 
CONTROL-PACKET-GENERATION  is  scheduled. 


When  this  event  is  performed,  the  user  first  checks  to 
see  if  it  is  active  or  not.  If  the  user  is  active,  then  it 
started  talking  since  the  time  at  which  the 
CONTROL-PACKET-GENERATION  was  scheduled,  and  no  further  action 
is  performed.  Otherwise,  a  control  packet  is  generated,  and 
its  FORWARD-CONTROL,  FEEDBACK-CONTROL  and  LENGTH  are  set,  as 
described  in  the  previous  section.  The  route  of  the  packet  is 
set  to  the  route  of  its  source,  its  GENERATION-TIME  is  set  to 
•TIME*,  and  TYPE  is  set  to  CONTROL.  The  packet  is  scheduled 
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to  arrive  at  the  first  link  of  its  route  at  •TIME*  + 
•PACKET-GENERATION-DELAY*,  by  adding  the  event  PACKET-ARRIVAL 


to  the  event-table.  Another  CONTROL-PACKET-GENERATION  is 
scheduled  for  the  appropriate  future  time. 

PACKET-ARRIVAL 

When  the  head  of  a  packet  arrives  at  a  link,  its 
ARRIVAL-TIME  is  set  to  •TIME*,  the  packet  is  placed  at  the  end 
of  the  queue,  and  link  queue  statistics  are  updated.  If  the 
link  is  idle  when  the  packet  arrives,  a  PACKET-TRANSMISSION  is 
scheduled  at  «TIME«  ♦  •PACKET-TRANSMISSION-DELAY*. 

PACKET-TAIL-ARRIVAL 

When  the  tail  of  a  voice  packet  arrives  at  a  link,  the 
link  checks  whether  the  packet's  source  is  on  its  list  of 
active  users.  If  not,  the  link  adds  the  source  to  its  list. 
Then  the  link  updates  its  stored  value  of  the  source's  rate, 
which  it  reads  from  the  packet.  When  the  tail  of  a  control 
packet  arrives,  PACKET-TAIL-ARRIVAL  does  nothing. 

PACM.IrIMHSHISS.IflM 

This  event  occurs  either  when  the  link  transmits  the  tail 
of  a  packet,  or  when  the  head  of  a  packet  arrives  at  an  empty 
queue.  If  the  link  is  transmitting  the  tail  of  a  packet  when 
this  event  is  performed,  it  will  update  its  records  of  the 
number  of  bits  transmitted.  If  the  packet  just  transmitted 
was  a  voice  packet  and  the  last  in  its  talk-spurt,  the  link 


will  remove  the  packet's  source  from  its  list  of  active  users 


Next,  whether  or  not  the  link  just  finished  a 
transmission,  it  checks  its  queue.  If  the  queue  is  empty,  the 
event  is  finished.  Otherwise,  the  link  gets  the  first  packet 
in  its  queue  and  changes  the  packet's  forward  control  value  as 
described  in  the  previous  section.  Then  the  link  schedules 
the  transmission  of  the  packet's  tail  at 

•TIME*  +  (PACKET-LENGTH/LINK-CAPACITY) 

+  *PACKET-TRANSMISSION-DELAY.  (5.3) 

Next  the  link  updates  relevant  statistics.  Finally,  the  link 
removes  itself  from  the  head  of  the  packet's  route  list,  and 
checks  for  the  packet's  next  destination.  If  the^  packet's 
route  is  empty,  its  next  destination  is  its  source's  partner, 
where  a  PACKET-ABSORPTION  is  scheduled  for 
•TIME*  ♦  LINK-PROPAGATION-DELAY 

♦  * PACKET- ABSORPTION-DELAY*.  (5.4) 

Otherwise,  the  head  of  the  packet  is  scheduled  to  arrive  at 

the  next  link  in  its  route  at 

•TIME*  +  LINK-PROPAGATION-DELAY 

+  *PACKET- ARRIVAL-DELAY*  (5.5) 

and  the  tail  of  the  packet  is  scheduled  to  arrive  at 

•TIME*  +  LINK-PROPAGATION-DELAY  ♦  *PACKET- ARRIVAL-DELAY* 

♦  (PACKET-LENGTH/LINK-CAPACITY).  (5.6) 


PACKET-ABSORPTION 

This  event  represents  the  arrival  of  a  packet  at  its 
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destination.  First,  the  packet's  net  delay  is  calculated  by 


DELAY=*TIME*  ♦  *PACKET- ABSORPTION-DELAY* 

-  PACKET-GENERATION-TIME.  (5.7) 
If  it  is  a  voice  packet,  the  user's  voice  packet  delay 
statistics  are  updated.  Otherwise,  the  user's  control  packet 
delay  statistics  are  updated.  The  user  then  sets  its  rate 
from  the  feedback  information,  and  stores  the  forward  control 
value  for  use  as  described  earlier.  If  the  packet  is  the  last 
in  its  talk-spurt,  a  VOICE-PACKET-GENERATION  is  scheduled  for 
the  packet's  destination. 

HIM IE 

When  an  UPDATE  is  performed  for  a  link,  the  link  first 
checks  whether  the  update  protocol  is  satisfied.  If  it  is, 
the  new  CONTROL  is  calculated  using  the  specified  update 
function,  and  another  UPDATE  is  scheduled  for  *TIME*  ♦ 
•UPDATE-INTERVAL*.  Otherwise,  another  UPDATE  is  scheduled  for 
•TIME*  +  *UPDATE-ATTEMPT- INTERVAL*. 

LINK-STATISTICS-COLLECTION 

This  event  collects  link  statistics,  and  adds  them  to  the 
•LINK-STAT-STREAM*,  which  is  output  to  a  file.  It  then  zeros 
the  statistics  and  schedules  the  next 
LINK-STATISTICS-COLLECTION  at  *TIME*  ♦ 
•LINK-STATISTICS-INTERVAL*. 


This  event  collects  user  statistics,  and  adds  them  to  the 
•USER-STAT-STREAM*,  which  is  output  to  a  file.  It  then  zeros 
the  statistics  and  schedules  the  next 
USER-STATISTICS-COLLECTION  at  *TIME«  ♦ 
•USER-STATISTICS- INTERVAL*. 

This  completes  the  description  of  the  simulation  program. 

5.3  Xh£  Network  Model 

In  order  to  have  a  basis  for  comparison,  we  chose  the 
same  network  model  used  by  Hayden.  This  network  is  a  scaled 
down  version  of  a  network  simulated  at  Lincoln  Laboratories 
[16],  The  network  consists  of  80  users  and  8  links,  and  its 
topology  is  illustrated  in  Figure  5.0.  The  original  network 
model  considered  traffic  flow  in  only  one  direction,  but  in 
order  to  model  the  effects  of  feedback  delay  we  must  consider 
two-way  traffic  flow.  In  order  to  use  this  model  for  two-way 
traffic  without  incorporating  additional  links,  we  view  all 
the  sources  as  being  at  the  same  location. 

We  make  the  same  user-partner  assignment  as  Hayden,  where 
each  user  i  has  as  its  partner  user  (8l-i),  for  is1,...,*»0. 
Ideally,  there  should  be  no  correlation  between  the  set  of 
links  in  a  user's  path  and  the  set  of  links  in  its  partner's 
path,  as  would  be  the  case  if  we  had  incorporated  extra  links 
to  handle  the  two-way  traffic.  However,  the  user  pairs 
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(32,49),  (31,50),  (30,51)  and  (29,52)  all  share  link  7.  But, 
since  they  constitute  only  8  out  of  24  users,  this  should  not 
be  a  serious  problem. 


The  program  as  described  in  the  previous  section  contains 
many  variables  related  to  the  actual  performance  of  a  physical 
network.  The  values  of  these  variables  are  given  in  Table 
5.1,  and  have  been  chosen  to  be  consistent  with  current 
technology. 


5.4 


Results 


In  this  section  we  describe  the  results  of  the 
simulation.  The  theory  of  the  preceding  chapters  addresses 
only  the  the  behavior  of  networks  with  fixed  configuration. 
In  practice,  however,  the  network  configuration  will  change 
rapidly  as  users  initiate  and  end  conversations.  The  ability 
of  an  algorithm  to  control  the  link  flows  in  a  changing 
network  depends  upon  the  rate  of  convergence  of  flows  in  a 
static  network.  If  the  time  required  for  the  controls  to 
converge  in  a  static  network  is  short  compared  to  the  rate  of 
change  of  the  dynamic  network,  the  algorithm  will  work  for  the 
dynamic  network.  Hence,  we  divide  our  results  into  two 
subsections:  static  results  and  dynamic  results. 


5.4.1  Stalls  Results 


LINK-CAPACITY 


40000  bits/sec 


LINK-PROPAGATION-DELAY  .003  sec 
•PACKET- ABSORPTION-DELAY*  .0005  sec 
•PACKET-ARRIVAL-DELAY *  .0005  sec 
•PACKET-TRANSMISSION-DELAY*  .0001  sec 
•PACKET-GENERATION-DELAY*  .0005  sec 

Network  Constants 


Table  5.2 
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In  all  of  the  experiments  described  in  this  section,  we 
studied  a  static  network  in  which  all  even  numbered  users  were 
active  for  the  entire  simulation.  For  this  network,  Hayden's 
fair  control  vector  is 
(8000, 8000, 8000, 8000, 8000, M000, 2667, 2000)  and  Jaffe's  fair 
control  vector  is  (4987, 4987,4987, 4987, 12050, 3555, 2461 ,1882) . 

Two  important  parameters  to  adjust  are  the  update 
interval  and  the  update  attempt  interval.  Preliminary  results 
indicated  that  the  update  attempt  interval  should  be  kept  as 
small  as  possible.  While  we  could  set  the  update  attempt 
interval  so  that  each  link  tries  to  update  each  time  it 
receives  a  packet,  this  would  slow  down  the  simulation 
considerably.  Instead,  we  set  the  update  attempt  interval  to 
20  msec.,  so  that  each  link  tries  to  update  after  receiving  a 
new  packet  from  each  of  its  users. 

Hayden  observed,  in  his  simulations,  that  setting  the 
update  interval  to  20  msec,  produced  severe  oscillations  in 
the  link  flows.  With  an  update  interval  of  100  msec.,  these 
oscillations  were  greatly  reduced.  For  comparison,  we  ran  our 
simulation  using  each  of  these  values. 

We  have  identified  three  parameters  to  adjust.  We  may 
choose  between  the  HAYDEN,  JAFFE  or  MOSELY  update  functions, 
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the  update  protocol  or  no  update  protocol,  and  fast  (every  20 
msec.)  or  slow  (every  100  msec.)  updates.  We  ran  the 
simulation  for  each  of  the  twelve  combinations. 

For  the  sake  of  brevity,  we  have  chosen  to  display  the 
results  for  links  2  and  5  only:  link  2  because  it  is  typical 
of  the  others,  and  link  5  because  it  is  atypical.  Link  5 
differs  from  the  rest  in  that,  when  the  rest  of  the  links' 
controls  correspond  to  Hayden's  fair  allocation,  link  5 
controls  none  of  its  users.  Thus  Hayden's  algorithm  will 
assign  link  5  a  control  equal  to  its  capacity.. 

Figures  5.1  -  5.3  show  link  2  controls  versus  time  for 
all  possible  combinations  of  update  function,  update  protocol 
and  rate  of  update.  Figures  5.4  -  5.6  show  the  same  data  for 
link  5.  Inspecting  these  figures,  we  make  the  following 
observations. 

Both  the  HAYDEN  and  JAFFE  update  functions  work 
moderately  well  with  slow  updates  on  link  2.  The  JAFFE  update 
function  can  also  control  link  5  with  slow  updates,  but  the 
HAYDEN  function  cannot.  Both  functions  seem  to  work 
marginally  better  with  update  protocols  than  without.  Neither 
function  works  at  all  well  with  fast  updates,  with  or  without 
the  protocol,  although  the  protocol  tends  to  damp  the 
oscillations  for  the  JAFFE  function.  These  results  are 
largely  what  we  expected,  based  on  Hayden's  simulations.  It 


is  a  little  surprising,  however,  in  view  of  the  theory  of 
Chapter  3,  that  the  JAFFE  function  performs  so  poorly  with 
fast  updates  and  the  update  protocol.  These  results  do  not 
contradict  the  theory,  though,  since  the  theory  makes  no 
claims  about  the  rate  of  convergence,  and  the  controls  do 
converge. 

The  MOSELY  update  function  is  capable  of  controlling  both 
links  well  under  all  circumstances.  Convergence  of  the 
controls  with  slow  updates  is  slightly  faster  without  the 
update  protocol  than  with.  Since  the  update  protocol  is  not 
necessary  for  the  controls  to  converge,  it  is  not  surprising 
that  the  protocol  slows  convergence  down,  as  the  protocol  must 
occasionally  prevent  a  possibly  beneficial  update. 

When  updates  are  fast,  there  is  almost  no  noticeable 
difference  between  the  performance  of  the  algorithm  with  or 
without  the  protocol.  For  link  2,  it  might  be  argued  that, 
because  the  controls  converge  to  the  fair  controls  from  below, 
the  update  protocol  is  nearly  always  satisfied,  and  hence  the 
performance  is  the  same  with  or  without  the  protocol.  In 
order  to  test  this  theory,  we  ran  the  simulation  with  the 
initial  controls  and  rates  set  high  (4000  bits/sec.)  to  try  to 
produce  a  control  sequence  that  converged  from  above.  The 
results  of  that  simulation  are  shown  in  Figures  5.7  -  5.8. 
The  controls  still  converge  from  below,  since  after  starting 
high,  the  link  cuts  its  control  sharply  to  limit  the  flow. 


There  is  more  difference  between  the  performance  with  and 
without  the  protocol  in  this  case,  but  the  difference  is  still 
small. 

From  these  figures  we  conclude  that  the  HAYDEN  and  JAFFE 
update  functions  work  best  with  slow  updates,  and  the  MOSELY 
function  works  best  with  fast  updates.  In  Figures  5.9  -  5.12, 
we  compare  the  performances  of  the  three  functions.  Since  the 
JAFFE  update  function  is  designed  to  converge  to  a  different 
fair  allocation  than  the  others,  it  is  difficult  to  assess  how 
well  it  does  this  relative  to  the  other  two  functions.  From 
Figures  5.9  and  5.10,  we  might  conclude  that  the  HOSELY 
function  performs  no  better  than  the  HAYDEN  or  JAFFE 
functions.  But  Figures  5.11  and  5.12  show  that  this  is  not 
so.  Collectively,  these  figures  seem  to  indicate  that  under 
best  case  conditions,  the  functions  work  approximately  equally 
well,  but  for  unusual  conditions,  the  MOSELY  function  works 
better. 

As  further  evidence  of  this  conclusion,  consider  Figures 
5.13  and  5.14.  These  figures  show  the  link  2  controls 
produced  by  the  HAYDEN  and  MOSELY  functions,  for  four 
different  sample  simulation  runs  for  each.  The  HAYDEN  update 
function  gives  varying  results  for  each  run,  while  the 
different  control  sequences  produced  by  the  MOSELY  function 
are  Indistinguishable.  The  variability  of  the  HAYDEN  function 
is  more  surprising  than  the  consistency  of  the  MOSELY 


function,  when  we  consider  how  little  difference  there  is  in 
the  loads  offered  to  the  network  for  each  sample  run.  For  the 
static  simulations,  the  pattern  of  conversations  does  not 
change,  only  the  order  of  arrival  of  the  packets  in  the  queues 
and  the  order  of  link  updates  differ  from  one  run  to  the  next. 

The  static  simulation  results  seem  to  indicate  that  the 
MOSELY  update  function  is  slightly  superior  to  the  others. 
However,  the  real  test  is  how  well  the  functions  perform  for  a 
dynamic  network. 

5.4.2  Dynamic  Results 

In  this  section  we  describe  the  results  of  the  dynamic 
simulation  experiments.  We  preface  this  discussion  by 
remarking  that  the  simulation  results  are  sufficiently 
unexpected  that  we  suspect  an  error  in  the  program,  and  we 
feel  that  additionaly  testing  is  called  for. 

For  all  the  simulation  runs  described  in  this  section,  we 
used  an  average  talk-spurt  length  of  60  packets,  corresponding 
to  a  1.2  second  talk-spurt  duration.  Thus,  the  link  loads 
change  rapidly,  as  one  user  stops  talking  and  its  partner 
begins.  We  ran  the  simulation  for  six  different  algorithms, 
using  each  of  the  three  update  functions  with  and  without  the 
update  protocol.  The  MOSELY  function  was  simulated  only  for 
fast  updates,  and  the  HAYDEN  and  JAFFE  functions  only  for  slow 


updates,  since  we  found  from  our  static  experiments  that  that 
is  how  these  functions  perform  best.  In  each  case,  we 
simulated  the  network  for  a  30  second  interval. 

For  the  sake  of  brevity,  we  show  results  only  for  links  2 
and  8.  By  the  symmetry  of  the  network,  links  1-4  are  all 
essentially  equivalent,  and  we  choose  link  2  as  representative 
of  the  others.  Link  8  is  the  most  heavily  loaded  link,  and  we 
choose  it  for  worst  case  behavior.  Figures  5.15  -  5.26  show, 
for  these  two  links  and  six  algorithms,  the  number  of  bits 
transmitted  over  a  0.10  second  interval,  along  with  the  number 
of  active  users  and  the  maximum  queue  size,  as  a  function  of 
time.  So  that  these  three  quantities  may  all  be  displayed  on 
the  same  graph,  we  have  scaled  the  number  of  users  up  by  a 
factor  of  100,  and  the  maximum  queue  size  by  a  factor  of  10. 

In  Figure  5.15  we  notice  that  for  times  between  20  and  25 
seconds,  the  flow  appears  to  exceed  the  link's  capacity  I  It 
only  appears  this  way,  however,  because  of  the  way  that  the 
links  count  the  bits  transmitted.  After  each  packet  is 
transmitted,  the  link  increments  its  bit  count  by  the  length 
of  the  packet.  This  count  is  zeroed  at  the  end  of  each  0.10 
second  interval.  Thus  a  packet  which  begins  transmission  in 
one  statistics  collection  interval,  and  finishes  in  another, 
will  count  as  having  been  wholly  tranmitted  in  the  second 
interval.  This  is  why  we  see  a  sequence  of  intervals  where 
the  flow  fluctuates  above  and  below  link  capacity. 


From  examining  these  figures  we  see  that  the  HAYDEN  and 
MOSELY  update  functions  produce  totally  unacceptable  queues. 
The  queues  for  the  JAFFE  function  are  much  smaller.  These 
results  are  unexpected  and  we  cannot  explain  them.  We  find 
the  size  of  the  queues  somewhat  surprising,  since  Hayden's 
simulations  rarely  showed  queues  exceeding  50  packets.  One 
possible  explanation  for  this  difference  is  that  Hayden  never 
ran  his  simulation  for  more  than  a  10  second  interval.  We 
notice  that  the  really  severe  queues  don't  generally  occur 
till  after  10  seconds.  This  could  be  due  to  the  fact  that  the 
simulation  starts  with  all  rates  and  controls  relatively  low, 
and  only  the  even  numbered  users  active.  Hence,  it  may  take 
several  seconds  for  these  effects  to  die  out  and  steady  state 
behavior  to  dominate.  Also,  Hayden  used  measured  flows  to 
calculate  the  controls  while  we  calculate  controls  using  a 
theoretical  flow  based  on  the  sum  of  the  users'  rates.  Since 
these  calculated  flows  do  not  reflect  the  presence  of  control 
packets  in  the  network,  the  resulting  controls  will  be 
somewhat  conservative.  This  seems  like  it  should  be  a  second 
order  effect,  however. 

In  general,  the  queues  build  up  in  response  to  sudden 
large  changes  in  the  number  of  active  users.  We  see  from  the 
static  results  that  it  takes  all  of  the  algorithms  around  2  - 
3  seconds  to  converge  to  the  correct  flow,  so,  when  the  number 
of  users  changes  rapidly,  the  algorithm  cannot  cope  with  the 
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change.  We  also  observe  that  the  queues  are  much  worse  for 
the  algorithm  with  protocol  than  without.  This  is  not 
surprising,  since  the  protocol  prevents  a  link  from  lowering 
its  control  until  all  its  users  are  sending  at  a  rate  lower 
than  its  control. 

It  is  interesting  to  note  that,  for  the  JAFFE  function, 
the  flows  seems  to  oscillate  more  in  response  to  the  changes 
in  the  number  of  active  users.  These  oscillations  are  similar 
to  the  oscillations  observed  for  the  static  simulations. 

Observing  that  the  JAFFE  function  produced  more  stable 
queues  than  the  other  functions,  and  that  the  MOSELY  function 
tended  to  result  in  higher  flows,  we  ran  the  simulation  for  a 
seventh,  hybrid  algorithm.  The  update  function  for  the 
algorithms  is  obtained  by  replacing  the  value  of  the  link 
control  wherever  it  appears  in  the  JAFFE  function  with  the 
value  of  the  maximum  user  rate.  The  hybrid  algorithm  was 
implemented  without  protocol  and  with  fast  updates.  The 
results  are  shown  in  Figures  5.27  and  5.28.  Unfortunately, 
there  is  no  distinct  improvement  in  performance  for  this 
algorithm. 

One  possible  explanation  for  the  difference  in  queue 
sizes  for  these  algorithms,  is  that  the  fair  flows  for  the 
JAFFE  function  are  smaller  than  the  fair  flows  for  the  other 
functions.  Hence,  we  expect  smaller  queues.  To  examine  this 


effect,  in  Figures  5.27  and  5.28,  we  plot  average  delay  versus 
average  flow  for  each  of  the  seven  algorithms.  The  averages 
are  computed  over  the  entire  30  second  interval.  While  the 
average  flows  are  indeed  smaller  for  the  JAFFE  and  HYBRID 
functions,  they  are  only  slightly  smaller.  The  difference  in 
average  delays,  however,  is  very  great,  and  we  are  forced  to 
conclude  that  the  JAFFE  function  gives  inherently  better  delay 
performance. 

We  believe  that  the  intrinsic  problem  with  all  the 
algorithms  thus  far  proposed  is  that  convergence  of  the 
controls  under  static  network  conditions  is  too  slow  when 
compared  with  the  rate  of  change  we  may  expect  in  a  dynamic 
network.  One  reason  for  this  is  the  long  feedback  delay 
between  the  links  and  the  sessions.  Two  things  could  be  done 
to  remedy  this.  Control  packets  could  be  generated  more 
often,  though  this  would  increase  overhead,  and  control 
packets  could  be  given  priority  in  the  queues. 

Another  reason  that  convergence  is  slow  is  that  the 
changes  that  a  link  makes  in  its  controls  are  always 
conservative.  Essentially,  the  links  assume  that  all  of  their 
users  will  be  affected  equally  by  changes  in  controls.  But 
this  assumption  is  clearly  erroneous.  For  example,  if  a  link 
decides  to  lower  its  control  from  p^  to  p2,  none  of  its  users 
that  are  currently  sending  at  rates  lower  than  p2  will  be 
affected.  Similarly,  if  a  link  raises  its  control  when  most 


-  133 


of  its  users  are  sending  at  much  less  than  the  current 
control,  it  is  unlikely  that  any  of  those  users  will  be 
affected  by  the  change.  With  the  HAYDEN  update  function,  the 
links  ignore  potential  feedback  information  from  the  users  by 
taking  note  of  only  the  sum  of  the  rates.  The  MOSELY  update 
function  makes  only  slightly  more  use  of  the  available 
information,  by  observing  the  sum  of  the  rates  and  the  maximum 
rates.  Perhaps  a  really  effective  update  function  could  be 
devised,  where  the  links  make  use  of  the  entire  rate  vector. 


Figure  5. 1 
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Figure  5.11 
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Figure  5.19 
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Figure  5.25 
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Flow/#  Users/Max.  Queue  for  Hybrid/no  protocol/fast  updates 
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Chapter  6 


Sug.gflS.tiQ tlS  1ST  Further  Research 

In  this  thesis,  we  have  considered  the  problem  of 
designing  a  distributed  fair  flow  control  algorithm  that  can 
be  implemented  asynchronously  and  remain  stable  in  the 
presence  of  feedback  delays.  After  developing  an  appropriate 
system  model,  we  analyzed  several  flow  control  algorithms.  We 
discovered  that  one  algorithm  which  is  unstable  when 
implemented  asynchronously,  can  be  made  stable  by  the  addition 
of  an  update  protocol  (the  generalized  link  memory  algorithm). 
One  algorithm  cannot  be  made  stable  even  with  the  update 
protocol  (Hayden),  and  another  is  stable  even  without  the 
update  protocol  (modified  Hayden).  This  last  algorithm  is 
particularly  interesting  in  that  the  links,  when  updating 
their  controls,  make  more  use  of  the  available  feedback 
information  than  the  other  algorithms. 

While  the  theoretical  results  indicate  that  these 
algorithms  should  perform  well,  the  results  are  only  valid  for 
a  static  network.  Computer  simulations  indicate  that  none  of 
the  algorithms  can  respond  to  changing  input  conditions  fast 
enough  to  effectively  control  a  dynamic  network.  Therefore, 
it  is  necessary  to  improve  the  response  time  of  the  system. 
This  might  be  accomplished  in  several  ways.  First,  queueing 


priority  should  ‘oe  given  to  the  control  packets,  to  speed  up 


convergence  of  the  controls.  Also,  when  links  become  very 
congested,  we  cignt  allow  the  system  to  discard  low  priority 
packets.  Finally,  as  described  at  the  end  of  Chapter  5 ,  we 
could  try  to  devise  better  update  functions  that  make  more 
complete  use  of  the  available  feedback  information. 

As  an  example,  we  propose  the  following  update  function: 
Pjn.1=jmax  r1J(tJ»).(oJ.f j")/Wj(a)  when 

(F(rj(tjn),cj)  when  fjn>cj  (6.1) 

where  a  is  some  appropriately  chosen  constant  such  that  0j£a<1, 

Wj(a)  is  the  number  of  sessions  whose  rate  is  higher  than 
a  max  «"ii(tin)»  r^t,11)  is  the  vector  of  rates 

( •  •  •  »rij(tjn)  i  • . . ) ,  and  F(Rj,Cj)  is  the  maximum  of  the 

coordinates  of  the  fair  allocation  over  the  set  defined  by 

rijiRij  (6.2.1) 

and 

(6.2.2) 

The  idea  behind  this  update  function  is  simple.  All  the 
update  functions  given  previously  change  the  controls  by 
conservative  amounts,  assuming  at  all  times  that  the  links 
control  all  their  users.  This  update  function  lets  the  link 
use  its  knowledge  of  the  rate  vector  to  make  better  estimates 
of  the  number  of  sessions  that  are  actually  under  its  control. 
When  the  flow  is  less  than  capacity,  the  link  uses  W.(a)  as  an 

w 

estimate  of  the  number  of  sessions  it  is  currently 
controlling.  When  the  flow  is  greater  than  capacity,  the  link 
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adopts  the  largest  control  that  would  let  each  of  its  sessions 
send  at  the  minimum  of  its  former  rate  and  the  new  control, 
while  guaranteeing  that  the  new  flow  would  be  less  than  or 
equal  to  capacity. 

In  conclusion,  we  remark  that  flow  control  algorithms  are 
just  one  example  of  many  different  distributed  asynchronous 
problems.  The  techniques  described  in  this  thesis,  that  is, 
the  use  of  update  protocols  and  more  complete  use  of  feedback 
information,  mignt  be  profitably  applied  to  other  problems  as 
well,  and  the  theorems  in  Chapter  4  might  be  used  to  analyze 
such  algorithms 


Appendix  A 


Theorem  1 J. .  If  d^t)  and  D± j(t )  satisfy  (3.10)  and  (3.11), 
then  for  any  t0  there  exists  tl^t^  such  that  for  all  t^t^ , 

t-dij(t)-Dik(t-dij(t)m°.  (A.1) 

for  all  jfe*,  i4i  jf  k€^i. 

Proof.  For  any  t0f  by  (3.11)  there  must  exist  Slirh  tMat. 

A  J 

Tij-Dij(Tijm°*  (A. 2) 

Let  Tis  max  Tjj.  By  (3.11),  there  must  also  exist  tjj  such 
that  ^ 

tij“dij(tij)>Ti. 

Let  tis  max  tii#  Then  for  t^t1,  by  (3.10), 

1**>**i 

t-dij(t)-Dik(t-dij(t)) 

-tij-dij^tij)-Oik(tij-dij(tij ) ) 

>Ti-Dik(Ti) 

-Tik-Dik<Tik) 

2t0. 

This  completes  the  proof  of  Theorem  3.2. 


(A. 3) 


(A. 4) 


Appendix  £ 


Lemma  3.2.1.  Let  Fj(z,)s(min  zk)(1-Wj/(2Wj))+cj/(2Wj)  and 
define  F(a)s(. . . tFj(2)f . . . ).  Let  z°s£  and  define  zn+1sj?(zn) . 

Then  z jn-*Zj«-p1+Wj/(2Wj) (cj/Wj-p1  >  and  2jn<cj/Wj  for  all  n. 


Proof.  First  we  show  that  2*  is  a  fixed  point  of  F.fO.  Note 

J 

that  since  plsmin  Cj/Wjf  Zj^p1  for  a11  J  and  P^=min  zk»^ 
Hence  "3  ^ 


Fj(Zj»)=p1(1-Wj/(2Wj))+Cj/(2Wj) 


=  Zj*' 


(B.  1  ) 


Now  let  !  I2 !  !=  max  |z.j,  we  show  that  F(*)  is  a  contraction 
under  this  norm. 

i |F(x)-F(y) i |=  max  i ( 1-Wj/(2wj) ) (min  xk-min  yk) | 

-  max  (  1-Wi/(2W.i) )  |  (min  xk-min  yk) 

-j  j  j  k  te. 

<  (1-  min  Wj/(2W:s))  max  ixk-yki 

1  « 

=  (1-  min  Wj/(2Wj))| Sx-2l i.  CB.2) 

Hence,  z*W2*. 


Now  suppose  2jn<cj/Wj  for  each  j,  for  some  n.  Then 
zJn.1<0-Wj/(2HJ))cj/HrcJ/(2HJ) 


*V"j- 


(B.3) 


Hence,  by  induction,  Zjn<cj/Wj  for  all  n.  This  completes  the 


proof  of  Lemma  3.2.1. 
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Theorem  3.4.  Let  S  be  a  linear  space  with  norm 


such 


that  {x:  I  !xi  lie}  is  compact  for  all  c.  Let  f:S-»S  and 

fn:S-*S  be  functions  such  that  fn->f  uniformly,  and  such  that 
i  jf  (x)  -  f  ( y )  I  i  <  i  S  x  -  yll  for  all  x,y€S.  Suppose  there 
exists  x*  such  that  x*sf(x*).  Define  xn+1-fn(Xn).  Then 

xn"^x*« 

Proof.  Without  loss  of  generality,  we  assume  x*=0.  For  each 


e>0,  define 


d(e)s  max  i  if (x) i  1. 
Hxli  <  e 


(C.1) 


The  maximum  must  exist  because  (x:  iixlij&c}  is  compact  for  all 
c.  Note  that  d(e)<e,  since  i  if (x) I i< I  |x  i  i . 


I  If (x)-f (e(x/ | Ixi i))i i<!  ix-e(x|  |x  I  i)  i  I 

ixi  i-e 


(C.2) 


I  I f C x ) j i < i ixi |-e+f(e(x/|  Jxi  i)) 
£ I  lx  I  i-(e-d(e) ) . 


(C.3) 


Now  let  e^-e/2  and  find  N  such  that 

I  lfn(x)-f(x) | ii(e1-d(e1 ))/2 
for  all  n^N.  Then 

-  170  - 


( C.  4  ) 


.v.--  , 


/ 


llxn!isllfnUn-1>ll 

<1  |f(xn-l)J  Mei-d(«i>)/2 

<  max  <elt i ixn-1 i  |-(e1-d(e1)))+(e1-d(ei))/2 

<  ®ax  (e1,  i  Un_i  I  !-(e1-d(e1))/2), 
where  the  third  step  follows  by  (C.3). 

So  for  any  xQ> 

•  1 1 <  max  (e,  i  !xq  !  i-(ei-d(ei ) )/2) , 

and  by  induction  on  n, 


(C.5) 


,\V 


'  ’ xn  •  i <  ®ax  (e,  S  SxqI  i-n(ei-d(e-j  ))/2). 
Thus,  for  any  e>0,  if  n>2 !  ixQ j |/(ei-d(ei ) ) , 
completes  the  proof  of  Theorem  3.4 


(C.6) 


(C.7) 


!xn !  !<e.  This 
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Define,  for  k=1,...,L, 

ek(0)=  aE 

(0.1.1) 

and 

E*<(0)  =  E 

(D. 1.2) 

where  a-  aJLn  1/W^  an(j  g  is  some  suitably  large 
that  £2pk/a.  Let 

constant  such 

e^ (n)=0 

(D.2.1 ) 

and 

El(n+l)=(1-a)E 

(D.2.2) 

for  all  n>1.  Now  define,  for  n>Nk 

ek(n+1)=(1-a)(ek(n)+Ek-1(n)) 

(D.3.1) 

El£(n+1  )s(  1-a)  (e^(n)+Ek(n) ) 

(D.3.2) 

and  for  Nkin<Nk+1 f  K>k, 

e^(n+1)=  pK«pk+ek(n+i ) 

(D.4.1) 

EK(n+1)sE, 

(D.4.2) 

where  N, <h2<. # .<NL,  Nlsl  and,  for  k>1, 
e“-1  (n)+Ek-1  (n)<(p»«-pk-1  )a/(1-a). 

(D.5) 

for  all  n>Nk# 

We  show  by  induction  that  the  sequence  Nj<N2<. . ,<Nl 
satisfying  (D.5)  exists,  and  that  the  sequences  (e^Cn)}  and 
{Ek(n)>  are  monotonically  non-increasing  and  converge  to  0, 


Furthermore,  we  show  that  the  sequences 

have  the 

properties  that,  for  k<K,  for  all  n, 

ek(nXeK(n) 

(D.6.1 ) 

Ek(n)iEK(n) 

(0.6.2) 

and 

pk-ek(n)^pK_eK(n) . 

(D.6.3) 

By  (D.2),  e1(n)=0  and  El(n)->0.  Hence, 

there 

must  exist 

a  time  N2  such  that  for  all  n>N2, 

el (nJ+E1 (n)<(p2-p1 )a/( 1-a) . 

(0.7) 

Suppose  we  have  (eK“1(n)}  and  (EK”l(n)}  such 

that 

e^-1  (n)-»0 

and  eK”1 (n)-*0.  Then  there  must  exist  a  time  N^  SUch  that, 
for  all  n>NK 

eK-1  (nJ+EK"1  (n)<(pK-pK-1  )a/(  1-a).  (D.8) 

Now  for  n>NKf 

eK(n+1 )s( 1-a) CeK(n)+EK”1 (n)).  (D.9) 

Since  (1-a)<1  and  EK-1(n)— >0,  eK(n)-*0.  Similarly,  eK(n)-*0 
implies  EK(n)-*0.  Hence,  we  have  shown  by  induction,  the 
existence  of  N.,  <n2<.  .  .<NL,  and  that  ek(n)-»0  and  Ek(n)-?0  for 
each  k. 

We  show  by  induction  that  e^(n)  and  EK(n)  are 
monotonically  non-increasing  for  each  k.  Clearly  e^Cn)  and 
E^Cn)  are  monotonically  non-increasing.  Now  suppose  that 
eK-1(n)  and  E^"^(n)  are  raonotonic  non-increasing  sequences. 
Then  eK(n+1 )<eK(n)  and  E^(n+1 )<EK(n)  for  n<N 

*K<Vl  >*<  1-a)(eK<NK)+EK’1  CNk)) 
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K.  For  n=NK, 


(D.10) 


Also, 


=  (1-a)(pK-pK“1+eK“1 CNk)+ek_1  (Nk) ) 
<( 1-a) (pK-pK-1 )+a(pK-pK”l ) 
=pK_pK-1 

«=K<»ic>. 

) 

EK(NK+i)s(l-a)(eK(NK)+EKCNK)) 

i(1-a)(eK(1)+£Ko)) 

=( 1-a) (pK-p1+E) 

<(1-a)(pL+E) 


=EK(Nk).  (D.11) 

The  next  to  last  step  holds  because  E  was  chosen  greater  than 
or  equal  to  pL/a. 

Now  suppose  eK(n)<eK(n-1 )  for  some  n>NK+i ,  Then 
eK(n+1)=(1-a)(eK<n)*EK-1<n)) 

i(1-a)(eKCn-1 )+EK_1  (n-1 ) ) 


=eK(n) . 


(D.12) 


Similarly,  if  EK(n)<EK(n-1 ) , 

EK(n+1)s(1-a)(eK(n)+EK(n)> 

i(1-a)(eK(n-1)+EK(n-1)) 

=EK(n).  (D. 1 3) 

Hence,  ek(n)  and  Ek(n)  are  monotonically  non-increasing  for 
all  k. 

Next  we  show  that  (D.6)  holds.  For  nsO,  ek(0)saE  and 
Ek(0)sE  for  all  k.  For  n=1,  ekC1)=pk-p1  and  Ek(1)*E.  Hence, 


(D.6)  holds  for  nsO  and  n=1 


We  show  that  (D.6.1)  and  CD. 6.2)  hold  by  induction. 
Suppose  that  (D.6.1)  and  (D.6. 2)  are  satisfied  for  some  n£1 • 
Let  J  be  the  largest  number  such  that  n>Nj.  we  show  that 
(D.6.1)  and  (D.6.2)  are  satisfied  for  n+ 1 ,  for  k<K.  We 
consider  three  cases:  k<K4J,  kiJ<K,  J<k<K. 

Let  k<K^J.  Then 
ek(n+1)=(1-a)(ek(n)+Ek“1(n)) 

4(1-a)(eK(n)+EK-1(n)) 

=eK(n+1 ) 

and 

Ek(n+1)=(1-a)(ek(n)+Ek(n)) 

<(1-a)(eK(n)+EK(n)) 

=EK(n+1). 

Let  k^J<K.  Then  by  ( D. 1 4) , 
ek(n+1 )^eJ (n+1 ) 

j£pK_pJ+eJ(n+i ) 

*eK(n+1)  (D. 1 6) 

and 

Ek(n+1)iEJ(n+1) 

iE 

=EK(n+1).  (D.17) 

Finally,  suppose  J<k<K.  Then 
ek(n+1 )  =  pk-p^+e^ (n+1 ) 


(D.  1 4) 


( D.  1 5) 


se^Cn+1 ) 


(D.18) 


and 

Ek(n+1)a£=EK(n+1).  CD. 1 9) 

So,  by  induction,  (D.6.1)  and  (D.6.2)  hold  for  all  n. 

Finally,  we  show  that  (D.6.3)  holds  for  all  n^l*  Let  J 
be  defined  as  above.  We  have  already  shown  in  (D.10),  for 
n>Nk 

ek(n).£ek(n+1 ) 

<pk_pk-1.  (D.20) 

Thus,  for  K£J, 

pk_ek(n)^pk 

ipK-1 

<pK_eK(n).  (D.21 ) 

For  J<K, 

pk»ek(n )^pJ-eJ ( n ) 

-pK_eK(n).  (D. 22) 

Therefore,  (D.6.3)  holds  for  all  n. 


This  completes  Appendix  D. 


Appendix  £ 


;;;  -*-  Mode:LISP;  Package :USER;  Base:10;  Fonts:MEDFNT  -*- 


Copyright  (c)  1984  by 

Jeannine  Mosely  and  Allan  C.  Wechsler 

It  is  the  intention  of  the  authors  that  this 
software  remain  in  the  public  domain,  and  that 
no  one  shall  impede  its  distribution,  nor 
distribute  it  for  profit. 


(DEFSTRUCT  (USER  .-CONC-NAME) 
PARTNER 
ROUTE 
(RATE  500) 

ID 

(PARTNER-RATE  1000) 


Another  user. 

A  list  of  links. 
Bits  per  second. 
A  number. 

Bits  per  second. 


The  following  six  components  are  statistics  that  we 


;;  reset  every  after  statistics  collection. 
(TOTAL-VOICE-PACKET-DELAY  0) 
(TOTAL-CONTROL-PACKET-DELAY  0) 

(NU MB ER-0F-V0ICE- PACKETS  0) 

(NUMBER-0F-C0NTR0L- PACKETS  0) 
(MAX-VOICE-PACKET-DELAY  0) 

(MAX-CONTROL-PACKET-DELAY  0) 

TALKING?  ;  T  or  NIL. 

PRINT-STATISTICS?  ;  T  or  NIL. 

) 


(DEFSTRUCT  (LINK  .-CONC-NAME) 
USERS 


(NUMBER-OF-USERS  0) 

QUEUE-FRONT 

QUEUE-BACK 


(QUEUE-LENGTH  0) 


;  An  a-list  of 
;  active  users  and 
;  their  rates. 

;  A  list  of  packets. 
;  The  last  vertebra 
;  of  QUEUE-FRONT. 

;  (Efficiency  hack.) 


::  Max  queue  length  is  reset  after  each  statistics 


;;  collection. 
(MAX-QUEUE-LENGTH  0) 

(CONTROL  1000) 

(CAPACITY  40000) 
(PROPAGATION-DELAY  0.003) 
PACKET-NOW-TRANSMITTING 

(NUMBER-OF-PACKETS-SENT  0) 
(T0TAL-NUMBER-0F-PACKETS-SENT 


;  Bits  per  second. 
;  Bits  per  second. 
;  Seconds. 

;  A  packet,  or  NIL 
;  if  link  idle. 

;  per  stats. 

)  :  ever. 


(NUMBER-OF-BITS-SENT  0) 
(TOTAL-NUMBER-OF-BITS-SENT  0) 
(TOTAL-PACKET-DELAY  0) 
(TOTAL-SQUARED-PACKET-DELAY  0) 
(PACKET-DELAY-HISTOGRAM 

(MAKE-ARRAY  11  * :TYPE  ' ART-16B) ) 
(PRINT-STATISTICS?  T) 

ID) 


;  per  stats. 
;  ever. 

;  ever. 

;  ever. 

;  ever. 

;  T  or  NIL 
;  A  number. 


(DEFSTRUCT  (PACKET  :C0NC-NAME) 
(FORWARD-CONTROL  1000000) 


FEEDBACK-CONTROL 


LENGTH 

ROUTE 

GENERATION-TIME 

ARRIVAL-TIME 

LAST-IN-TALK-SPURT? 

TYPE 

SOURCE 

SOURCE- RATE 

DESTINATION) 


Minimum  control 
seen  so  far  on 
this  traverse. 
Control  data 
going  back  to 
starting  point. 


VOICE  or  CONTROL. 


(DEFSTRUCT  (EVENT  ;C0NC-NAME) 
FUNCTION 
TIME 

ARGUMENTS) 


;;;  Global  variables. 


(DECLARE  (SPECIAL  *USERS* 

•LINKS* 

•TIME* 

•USER-STAT-STREAM* 

•LINK-STAT-STREAM* 

•EVENT-TABLE* 

•NEXT-EVENT-NUMBER*) ) 


All  users. 
All  links. 
Simulated. 


;  The  Heap  of 
;  Things  to  Come. 

;  Index  into  heap. 


(DEFC0NST  • PACKET- ABSORPTION- DELAY*  0.0005) 
(DEFCONST  *PACKET-ARRIVAL-DELAY •  0.0005) 
(DEFCONST  * PACKET-TRANSMISSION-DELAY*  0.0001) 
(DEFCONST  *PACKET-GENERATION-DELAY •  0.0005) 


(DEFCONST  *AVERAG E-TALK-SPURT-LENGTH*  60) 
(DEFCONST  *USERS-TALK- FOR EVER*  T) 

(DEFCONST  *CONTROL- PACKET-SPACING*  0.10) 


In  packets. 
Infinite  talk 
spurts?  T  or  NIL 
Seconds. 


(DEFCONST  «UPDATE- INTERVAL*  0.10) 


:  Seconds 


•  _» 


(DEFCONST  •UPDATE-ATTEMPT-INTERVAL*  0.02)  ;  Seconds. 

(DEFCONST  ^UPDATE- PROTOCOL*  'MOSELY)  ;  HAYDEN  or  MOSELY, 

(DEFCONST  «UPDATE-FUNCTION*  * MOSELY-UPDATE-FUNCTION) 

(DEFCONST  *LINK-STATISTICS- INTERVAL*  0.1) 

(DEFCONST  *USER-STATISTICS- INTERVAL*  0.5) 

(DEFCONST  *USERS-T0- PRINT* 

•(1  9  15  19  21  29  35  39  41  49  55  59  61  69  75  79)) 


;;;  Top  level  function. 

(DEFUN  RUN-NETWORK  (TIME-LIMIT) 

(INITIALIZE) 

(WITH-OPEN-FILE 

(•USER-STAT-STREAM*  "oz : < j9>user-stats. text"  ' :OUT) 
(WITH-OPEN-FILE 

(•LINK-STAT-STREAM*  "oz : <J9>1 ink-stats, text"  • :OUT) 
(FORMAT  *USER-STAT-STREAM*  " 

User  ID  Time  AVD  MVD  ACD 

MCD  Rate  Feedback”) 

(FORMAT  *LINK-STAT-STREAM*  " 

Link  ID  Time  #  Bits  Control  #  Users 

Max.  Q  Q  it  Packets") 

(SETQ  ‘TIME*  -1) 

(SIMULATE  TIME-LIMIT) 

( PRINT-LINK-HISTOGRAMS) ) ) ) 

;;;  Network  initialization. 

(DEFUN  INITIALIZE  () 

(SETQ  *USERS*  NIL) 

(CLEAR-EVENT-TABLE) 

(LET  ((LINK-1  (MAKE-LINK  ID  1 ) ) 

(LINK-2  (MAKE-LINK  ID  2  PRINT-STATISTICS?  T) ) 

(LINK-3  (MAKE-LINK  ID  3)) 

(LINK-4  (MAKE-LINK  ID  4)) 

(LINK-5  (MAKE-LINK  ID  5)) 

(LINK-6  (MAKE-LINK  ID  6)) 

(LINK-7  (MAKE-LINK  ID  7)) 

(LINK-8  (MAKE-LINK  ID  8  PRINT-STATISTICS?  T))) 

(SETQ  *LINKS*  (LIST  LINK-1  LINK-2  LINK-3  LINK-4 

LINK-5  LINK-6  LINK-7  LINK-8)) 
(USERS  '((1  8)  (21  28)  (41  48)  (61  68))  LINK-8) 
(USERS  '((9  14)  (29  34)  (49  54)  (69  74))  LINK-7) 
(USERS  '((15  18)  (35  38)  (55  58)  (75  78))  LINK-6) 
(USERS  '((19  20)  (39  40)  (59  60)  (79  80))  LINK-5) 

(USERS  '((1  20))  LINK-1) 

(USERS  '((21  40))  LINK-2) 

(USERS  '((41  60))  LINK-3) 
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(USERS  '((61  80))  LINK-4) ) 

( ESTABLISH-USER- PARTNERS) 

( INITIAL IZE-USERS-TO-PRINT) ) 

(DEFUN  INITIAL IZE-USERS-TO-PRINT  () 

(DOLIST  (USER-TO-PRINT  "USERS-TO-PRINT") 

(SETF  (USER-PRINT-STATISTICS? 

(FIND-KNOWN-USER  USER-TO-PRINT))  T))) 

(DEFUN  USERS  (ID-RANGES  LINK) 

(LOOP  FOR  (LOW-ID  HIGH-ID)  IN  ID-RANGES 
DO 

(LOOP  FOR  ID  FROM  LOW-ID  TO  HIGH-ID 
DO 

(ADD-OR-MODIFY-USER  ID  LINK)))) 

(DEFUN  ADD-OR-MODIFY-USER  (ID  LINK) 

(LET  ((USER  (FIND-USER  ID))) 

(PUSH  LINK  (USER-ROUTE  USER)))) 

(DEFUN  ESTABLISH-USER-PARTNERS  () 

(LOOP  FOR  USER  IN  "USERS* 

DO 

(SETF  (USER-PARTNER  USER) 

(FIND-KNOWN-USER  (-  81  (USER- ID  USER)))))) 

(DEFUN  FIND-USER  (ID) 

(OR  (FIND-KNOWN-USER  ID) 

(LET  ((USER  (MAKE-USER  ID  ID))) 

(PUSH  USER  "USERS*) 

USER))) 

(DEFUN  FIND-KNOWN-USER  (ID) 

(LOOP  FOR  USER  IN  "USERS" 

DO 

(WHEN  (  =  (USER-ID  USER)  ID) 

(RETURN  USER)))) 

;;;  Event  table  hackery. 

(DEFUN  CLEAR-EVENT-TABLE  () 

(SETQ  "EVENT- TABLE"  (MAKE-ARRAY  2048)) 

(SETQ  "NEXT-EVENT-NUMBER"  1)) 

(DEFUN  ADD-EVENT- TO-HEAP  (EVENT) 

(PERCOLATE-UP  EVENT  "NEXT-EVENT-NUMBER") 

(INCF  "NEXT-EVENT-NUMBER")) 

(DEFUN  GET-NEXT-EVENT  () 

(WHEN  (>  "NEXT-EVENT-NUMBER"  1) 

(LET  ((EVENT  (AREF  "EVENT-TABLE"  1)) 

(HOLE  (PERCOLATE-DOWN  1))) 

(UNLESS  (=  HOLE  (-  "NEXT-EVENT-NUMBER"  1)) 


(PERCOLATE-UP  (AREF  *EVENT-TABLE* 

(-  *NEXT-EV ENT- NUMBER*  1)) 

HOLE) ) 

(DECF  *NEXT-EVENT-NUMBER*) 

EVENT))) 

(DEFUN  PERCOLATE-UP  (EVENT  INDEX) 

(LET  ((PARENT-INDEX  (LSH  INDEX  -1))) 

(LET  ((PARENT-EVENT  (AREF  *EVENT-TABLE* 

PARENT-INDEX))) 

(IF  (OR  (s  PARENT-INDEX  0) 

(EVENTS- IN-ORDER  PARENT-EVENT 

EVENT)) 

;;  EVENT  goes  here  —  put  it  here. 

(SETF  (AREF  *EVENT-TABLE*  INDEX)  EVENT) 

;;  EVENT  goes  higher  — 

; ;  put  parent  here  and  recurse. 

(SETF  (AREF  *EVENT-TABLE*  INDEX) 

PARENT—  EVENT ) 

(PERCOLATE-UP  EVENT  PARENT-INDEX))))) 

(DEFUN  PERCOLATE-DOWN  (INDEX) 

(LET  ((LEFT-CHILD-INDEX  (LSH  INDEX  1))) 

(IF  (  LEFT-CHILD-INDEX  *NEXT- EVENT- NUMBER*) 

INDEX 

(LET  ((RIGHT-CHILD-INDEX  (♦  1  LEFT-CHILD-INDEX)) 
(LEFT-CHILD  (AREF  *EVENT-TABLE* 

LEFT-CHILD-INDEX))) 

(IF  (<  RIGHT-CHILD-INDEX  *NEXT-EVENT-NUMBER») 

(LET  ((RIGHT-CHILD  (AREF  »EVENT- TABLE* 

RIGHT-CHILD-INDEX))) 
(COND  ( (EVENTS-IN-ORDER  LEFT-CHILD 

RIGHT-CHILD) 

(SETF  (AREF  *EVENT-TABLE*  INDEX) 

LEFT— CH ILD ) 

( PERCOLATE-DOWN  LEFT-CHILD-INDEX) ) 

(T 

(SETF  (AREF  *EVENT-TABLE*  INDEX) 
RIGHT-CHILD) 

( PERCOLATE-DOWN  RIGHT-CHILD-INDEX) ) ) ) 
(SETF  (AREF  *EVENT-TABLE*  INDEX) 

LEFT-CHILD) 

LEFT-CHILD-INDEX))))) 

(DEFUN  EVENTS-IN-ORDER  (El  E2) 

(LET  ( (T1  (EVENT-TIME  ED) 

(T2  (EVENT-TIME  E 2))) 

(OR 

(<  T1  T 2) 

(AND 

(«  T1  T2) 


m 
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(LET  ((FI  (EVENT-FUNCTION  ED) 

(F 2  (EVENT-FUNCTION  E2))) 

(OR  (AND  (EQ  FI  I'LINK-STATISTICS-COLLECTION) 

(NOT  (EQ  F2  #' LINK-STATISTICS-COLLECTION) ) ) 
(AND  (EQ  FI  #' LINK-STATISTICS- COLLECTION) 

(EQ  F2  #• LINK-STATISTICS-COLLECTION) 

(<  (LINK-ID  (CAR  (EVENT-ARGUMENTS  ED)) 

(LINK-ID  (CAR  (EVENT-ARGUMENTS  E2))))) 

(AND  (EQ  FI  #’ USER-STATISTICS- COLLECT ION) 

(NOT  (EQ  F2  #' USER-STATISTICS-COLLECTION) ) 
(NOT  (EQ  F2 

# • LINK-STATISTICS-COLLECTION) ) ) 
(AND  (EQ  FI  #' USER-STATISTICS- COLLECTION) 

(EQ  F2  #' USER-STATISTICS-COLLECTION) 

(<  (USER-ID  (CAR  (EVENT-ARGUMENTS  ED)) 

(USER-ID  (CAR  (EVENT- ARGUMENTS  E2) )))))) 

)))) 

; ; ;  The  Guts. 

(DEFUN  SIMULATE  (TIME-LIMIT) 

(EVENT  #• SIMULATION-STARTUP  0) 

(LOOP  FOR  EVENT  s  (GET-NEXT-EVENT) 

WHILE  EVENT 

UNTIL  (<  TIME-LIMIT  (EVENT-TIME  EVENT)) 

DO 

(PERFORM- EVENT  EVENT))) 

(DEFUN  PERFORM-EVENT  (EVENT) 

(SETQ  “TIME*  (EVENT-TIME  EVENT)) 

(LEXPR- FUN CALL  (EVENT-FUNCTION  EVENT) 

(EVENT-ARGUMENTS  EVENT))) 


Events. 


;;;  Schedule  an  event 

(DEFUN  EVENT  (EVENT- FUNCTION  TIME  AREST  EVENT-ARGUMENTS) 
(WHEN  (<  TIME  *TIME*) 

(FERROR  "Tried  to  schedule  event  in  the  past.")) 
(ADD-EVENT-TO-HEAP 
(MAKE-EVENT  TIME  TIME 

FUNCTION  EVENT-FUNCTION 

ARGUMENTS  (COPYLIST  EVENT-ARGUMENTS)))) 


;;;  Everything  starts  up. 


(DEFUN  SIMULATION-STARTUP  () 
(LOOP  FOR  USER  IN  "USERS* 


V/.-.V.V. 


(IF  (EVENP  (USER-ID  USER)) 

(START- TALKING  USER) 

(EVENT  #• CONTROL-PACKET-GENERATION 

(+  *TIME*  *CONTROL- PACKET- SPACING*) 

USER)) 

( EVENT  #• USER-STATISTICS-COLLECTION 

(+  *TIME*  *USER-STATISTICS- INTERVAL*) 

USER) ) 

(LOOP  FOR  LINK  IN  *LINKS* 

DO 

(EVENT  #• UPDATE 
(+  *TIME* 

(SI :RANDON-IN- RANGE  0  *UPDATE-INTERVAL») ) 
LINK) 

( EVENT  # • LINK-STATISTICS-COLLECTION 

(+  *TIME«  *L INK- STATISTICS- INTERVAL*) 

LINK))) 

;;;  A  link  begins  passing  the  first  packet  in  its  queue  to 
;;;  the  next  link  in  that  packet's  route. 

(DEFUN  PACKET-TRANSMISSION  (LINK) 

(LET  ((LAST- PACKET-SENT 

(LINK-PACKET-NOW-TRANSMITTING  LINK))) 

(WHEN  LAST-PACKET-SENT 

(INCF  (LINK-NUMBER-OF-BITS-SENT  LINK) 

(PACKET-LENGTH  LAST-PACKET-SENT) ) 

(INCF  (LINK-TOTAL-NUMBER-OF-BITS-SENT  LINK) 
(PACKET-LENGTH  LAST-PACKET-SENT) ) 

(IF  (PACKET-LAST-IN-TALK-SPURT?  LAST- PACKET-SENT) 

( REMOVE-USER-FROM-LINK 
(PACKET-SOURCE  LAST- PACKET-SENT) 

LINK)))) 

(LET  ((PACKET  (POP  (LINK-QUEUE-FRONT  LINK)))) 

(SETF  (LINK-PACKET-NOW-TRANSMITTING  LINK)  PACKET) 

(UNLESS  (NULL  PACKET) 

(INCF  (LINK-NUMBER-OF-PACKETS-SENT  LINK)) 

(INCF  (LINK-TOTAL-NUMBER-OF-PACKETS-SENT  LINK)) 

(DECF  (LINK-QUEUE-LENGTH  LINK)) 

(LET  ((PACKET-DELAY  (-  *TIME* 

(PACKET-ARRIVAL-TIME  PACKET)))) 
(INCF  (LINK-TOTAL-PACKET-DELAY  LINK)  PACKET-DELAY) 
(INCF  (LINK-TOTAL-SQUARED-PACKET-DELAY  LINK) 

(*  PACKET-DELAY  2)) 

(INCF  (AREF  (LINK-PACKET-DELAY-HISTOGRAM  LINK) 

(MIN  10 

(FIX  (//  PACKET-DELAY  0.002)))))) 
(SETF  (PACKET- FORWARD-CONTROL  PACKET) 

(MIN  ( PACKET-FP RWARD-CONTROL  PACKET) 
(LINK-CONTROL  LINK))) 

(EVENT  #' PACKET-TRANSMISSION 
(♦  *TIME* 
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(//  (PACKET-LENGTH  PACKET) 

(FLOAT  (LINK-CAPACITY  LINK))) 
•PACKET-TRANSMISSION-DELAY • ) 

LINK) 

(LET  ((DESTINATION  (POP  (PACKET-ROUTE  PACKET)))) 
(IF  (NULL  DESTINATION) 

(EVENT  #» PACKET- ABSORPTION 
U  *TIME* 

(LINK-PROPAGATION-DELAY  LINK) 
•PACKET- ABSORPTION-DELAY*) 

PACKET 

(PACKET-DESTINATION  PACKET)) 

(EVENT  #> PACKET- ARRIVAL 
(+  niME* 

(LINK-PROPAGATION-DELAY  LINK) 
•PACKET-ARRIVAL-DELAY*) 

PACKET 

DESTINATION) 

(EVENT  #• PACKET-TAIL-ARRIVAL 
(+  *TIME* 

(//  (PACKET-LENGTH  PACKET) 

(FLOAT  (LINK-CAPACITY  LINK))) 
(LINK- PROPAGATION-DELAY  LINK) 
•PACKET- ARRIVAL-DELAY*) 

PACKET 

DESTINATION)))))) 

;;;  A  packet  Is  received  by  its  intended  target  user. 

(DEFUN  PACKET-ABSORPTION  (PACKET  USER) 

(LET  ((DELAY  (-  (+  •TIME*  *PACKET- ABSORPTION-DELAY*) 

( PACKET-GENERATION-TIME  PACKET) ) ) ) 
(SELECTQ  (PACKET-TYPE  PACKET) 

(VOICE 

( INCF  (USER-TOTAL-VOICE-PACKET-DELAY  USER)  DELAY) 
( INCF  ( USER-NU MB ER-OF- VOICE- PACKETS  USER)) 

(SETF  (USER-MAX-VOICE-PACKET-DELAY  USER) 

(MAX  (USER-MAX-VOICE-PACKET-DELAY  USER) 
DELAY))) 

(CONTROL 

(INCF  (USER-TOTAL-CONTROL-PACKET-DELAY  USER) 
DELAY) 

(INCF  (USER-NUMBER-OF-CONTROL-PACKETS  USER)) 

(SETF  (USER-MAX-CONTROL-PACKET-DELAY  USER) 

(MAX  (USER-MAX-CONTROL-PACKET-DELAY  USER) 
DELAY))))) 

(SETF  (USER-RATE  USER) 

(PACKET- FEEDBACK-CONTROL  PACKET) ) 

(SETF  (USER-PARTNER-RATE  USER) 

(PACKET-FORWARD-CONTROL  PACKET)) 

(WHEN  (PACKET-LAST-IN-TALK-SPURT?  PACKET) 

(SETF  (USER-TALKING?  ( USER- PARTNER  USER))  NIL) 
(START-TALKING  USER))) 


VV\ 

v. 


(DEFUN  REMOVE-USER-FROM-LINK  (USER  LINK) 
(SETF  (LINK-USERS  LINK) 

(DELQ  (ASSQ  USER  (LINK-USERS  LINK)) 
(LINK-USERS  LINK))) 

(DECF  (LINK-NUMSER-OF-USERS  LINK))) 


;;;  This  is  a  separate  function  so  it  can  be  called  at 
;;;  initialization  time. 

(DEFUN  START-TALKING  (USER) 

(SETF  (USER- TALKING?  USER)  T) 

(EVENT  #• VOICE-PACKET-GENERATION 
(+  *TIME* 

•PACKET- ABSORPTION-DELAY  * 
(RANDOM-INTER-TALK-SPURT-SILENCE)) 

USER)) 

; ; ;  A  packet  begins  to  arrive  at  a  link. 

(DErUN  PACKET-ARRIVAL  (PACKET  LINK) 

(SETF  (PACKET-ARRIVAL-TIME  PACKET)  •TIME*) 

;;  Enqueue  packet. 

(LET  ((OLD-QUEUE-FRONT  (LINK-QUEUE-FRONT  LINK))) 

(LET  ((NEW-QUEUE-SACK  (LIST  PACKET))) 

(IF  (NULL  (LINK-QUEUE-FRONT  LINK)) 

(SETF  (LINK-QUEUE-FRONT  LINK)  NEW-QUEUE-BACK) 
(RPLACD  (LINK-QUEUE-BACK  LINK) 
NEW-QUEUE-BACK)) 

(SETF  (LINK-QUEUE-BACK  LINK) 

NEW-QUEUE-BACK) 

( INCF  (LINK-QUEUE-LENGTH  LINK)) 

(SETF  (LINK-MAX-QUEUE-LENGTH  LINK) 

(MAX  (LINK-QUEUE-LENGTH  LINK) 

(LINK-MAX-QUEUE-LENGTH  LINK)))) 

;;  If  link  is  idle,  schedule  instant  transmission. 
(UNLESS  (OR  (LINK-PACKET-NOW-TRANSMITTING  LINK) 

(NOT  (NULL  OLD-QUEUE-FRONT))) 

(EVENT  #> PACKET- TRANSMISSION 

(♦  *TIME*  •PACKET-TRANSMISSION-DELAY*) 
LINK)))) 

;;;  The  tail  end  of  a  packet  arrives  at  a  link. 

(DEFUN  PACKET-TAIL-ARRIVAL  (PACKET  LINK) 

(WHEN  (EQ  (PACKET-TYPE  PACKET)  » VOICE) 

(LET  ((ACTIVE-USER  (ASSQ  (PACKET-SOURCE  PACKET) 

(LINK-USERS  LINK)))) 

(IF  ACTIVE-USER 

(RPLACD  ACTIVE-USER 

(PACKET-SOURCE-RATE  PACKET)) 
(ADD-USER-TO-LINK 
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(PACKET-SOURCE  PACKET) 

LINK 

(PACKET-SOURCE- RATE  PACKET)))))) 


(DEFUN  ADD-USER-TO-LINK  (USER  LINK  PACKET-LENGTH) 

(PUSH  (CONS  USER  PACKET-LENGTH) 

(LINK-USERS  LINK)) 

(INCF  (LINK-NUMBER-OF-USERS  LINK))) 

;;;  A  user  creates  a  voice  packet. 

(DEFUN  VOICE-PACKET-GENERATION  (USER) 

(LET  ((PACKET 

(MAKE- PACKET 

FEEDBACK-CONTROL  ( USER- PARTNER-RATE  USER) 

LENGTH  (FIX  (//  (USER-RATE  USER)  50)) 

ROUTE  (USER-ROUTE  USER) 

GENERATION-TIME  *TIME* 

LAST-IN-TALK-SPURT? 

(IF  *USERS-TALK-FOR  EVER  * 

NIL 
(=  0 

(RANDOM 

•AVERAGE-TALK-SPURT-LENGTH*)  )  ) 

TYPE  'VOICE 
SOURCE  USER 

SOURCE-RATE  (USER- RATE  USER) 

DESTINATION  (USER-PARTNER  USER)))) 

(LET  ((FIRST-LINK  (POP  (PACKET-ROUTE  PACKET)))) 

(EVENT  #• PACKET-ARRIVAL 

(+  *TIME*  *PACKET-GENERATION-DELAY*) 

PACKET 

FIRST-LINK) 

(EVENT  #• PACKET- TAIL-ARRIVAL 

(+  *TIME*  *PACKET-GENE RATION-DELAY*) 

PACKET 

FIRST-LINK)) 

(IF  ( PACKET-LAST-IN-TALK-^PURT?  PACKET) 

(EVENT  #' CONTROL-PACKET-GENERATION 

(♦  *TIME*  ^CONTROL- PACKET- SPACING*) 

USER) 

(EVENT  #' VOICE- PACKET-GENE RATION 

(♦  *TIME*  (SI :RANDOM-IN-RANGE  0.018  0.022)) 
USER)))) 

;;;  A  user  creates  a  control  packet. 

(DEFUN  CONTROL-PACKET-GENERATION  (USER) 

(UNLESS  (USER-TALKING?  USER) 

(LET  ((PACKET 

(MAKE-PACKET  FEEDBACK-CONTROL 

(USER-PARTNER-RATE  USER) 

LENGTH  10 
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ROUTE  (USER-ROUTE  USER) 
GENERATION-TIME  *TIME* 

TYPE  'CONTROL 
SOURCE  USER 

DESTINATION  ( USER- PARTNER  USER)))) 
(EVENT  #' PACKET-aRRIVAL 

(+  *TIME*  *PACKET-GENERATION-DELAY*) 

PACKET 

(POP  (PACKET- ROUTE  PACKET))) 

(EVENT  #• CONTROL-PACKET-GENERATION 

(+  *TIME*  *CONTROL- PACKET- SPACING*) 

USER)))) 

;;;  A  link  updates  its  control. 

(DEFUN  UPDATE  (LINK) 

(MULTIPLE- VALUE-BIND  (MAX-RATE  FLOW) 
(MAXIMIZE-AND-SUM-RATES-ON  LINK) 

(IF  (OR  (SELECTQ  *UPDATE-PROTOCOL* 

(MOSELY  (NOT  (  MAX-RAT; 

(LINK-CONTROL  LINK)))) 

(HAYDEN  NIL)) 

(ZEROP  (LINK-NUMBER-OF-USERS  LINK))) 

(EVENT  #' UPDATE 

(+  *TIME*  *UPDATE- ATTEMPT- INTERVAL*) 

LINK) 

(SETF  (LINK-CONTROL  LINK) 

(FUNCALL  *UPDATE-FUNCTION* 

LINK  MAX-RATE  FLOW)) 

(EVENT  #' UPDATE 

(+  *TIME*  *UPDATE- INTERVAL*) 

LINK)))) 

(DEFUN  MAXIMIZE-AND-SUM-RATES-ON  (LINK) 

(LOOP  FOR  (USER  .  RATE)  IN  (LINK-USERS  LINK) 

MAXIMIZE  RATE  INTO  MAX-RATE 
SUMMING  RATE  INTO  SUM 
FINALLY 

(RETURN  MAX-RATE  SUM))) 

(DEFUN  MOSELY-UPDATE-FUNCTION  (LINK  MAX-RATE  FLOW) 

(LET  ((ALPHA  1.0) 

(EFFECTIVE-CAPACITY  (*0.8  (LINK-CAPACITY  LINK)))) 
(MAX  (MIN  (+  MAX- RATE 

(//  (*  ALPHA  (-  EFFECTIVE-CAPACITY  FLOW)) 
(LINK-NUMBER-OF-USERS  LINK))) 
EFFECTIVE- CAPACITY) 

(//  EFFECTIVE-CAPACITY 

(LINK-NUMBER-OF-USERS  LINK))))) 

(DEFUN  HAYDEN-UPDATE-FUNCTION  (LINK  MAX-RATE  FLOW) 

MAX-RATE 

(LET  ((ALPHA  1.0) 


(EFFECTIVE-CAPACITY  (*0.8  (LINK-CAPACITY  LINK)))) 
(MAX  (MIN  (  +  (LINK-CONTROL  LINK) 

(//  (»  ALPHA  (-  EFFECTIVE-CAPACITY  FLOW)) 
(LINK-NUMBER-OF-USERS  LINK))) 
EFFECTIVE-CAPACITY) 

(//  EFFECTIVE-CAPACITY 

(LINK-NUMBER-OF-USERS  LINK))))) 

(DEFUN  JAFFE-UPDATE-FUNCTION  (LINK  MAX-RATE  FLOW) 

MAX-RATE 

(LET  ((ALPHA  1.0) 

(EFFECTIVE-CAPACITY  (*  0.8  (LINK-CAPACITY  LINK))) 
(CONTROL  (LINK-CONTROL  LINK))) 

(MAX  (MIN  (+  CONTROL 

(//  (•  ALPHA  (-  EFFECTIVE-CAPACITY 
FLOW 

CONTROL)) 

(+  (LINK-NUMBER-OF-USERS  LINK)  1))) 
EFFECTIVE-CAPACITY) 

(//  EFFECTIVE-CAPACITY 

(+  (LINK-NUMBER-OF-USERS  LINK)  1))))) 

;;;  Some  network  behavior  statistics  are  recorded. 

(DEFUN  LINK-STATISTICS-COLLECTION  (LINK) 

(COND  ((LINK-PRINT-STATISTICS?  LINK) 

(FORMAT  *LINK-STAT-STREAM* 

*  t  1 OD  2,1,10$  10D  1 OD  1 OD  10D  1 OD  10D" 
(LINK-ID  LINK) 

•TIME* 

(LINK-NUMBER-OF-BITS-SENT  LINK) 

(FIXR  (LINK-CONTROL  LINK)) 
(LINK-NUMBER-OF-USERS  LINK) 
(LINK-MAX-QUEUE-LENGTH  LINK) 
(LINK-QUEUE-LENGTH  LINK) 
(LINK-NUMBER-OF-PACKETS-SENT  LINK) ) 
(ALTER-LINK  LINK 

NUMBER-OF-BITS-SENT  0 
MAX-QUEUE-LENGTH  0 
NUMBER-OF-PACKETS-SENT  0))) 

(EVENT  #' LINK-STATISTICS- COLLECT ION 

(+  *TIME*  *L INK- STATISTICS- INTERVAL*) 

LINK)) 

(DEFUN  USER-STATISTICS-COLLECTION  (USER) 

(COND 

((USER-PRINT-STATISTICS?  USER) 

(FORMAT 

•USER-STAT-STREAM* 

"  %  1 OD  2,1,10$  2,1,10$  2,1,10$  2,1,10$  2,1,10$  10D  10D" 
(USER-ID  USER) 

•TIME* 

(SAFE-//  (USER- TOTAL- VOICE-PACKET-DELAY  USER) 
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(  U SER- NUMB ER-OF- VO ICE- PACKETS  USER) ) 
(USER-MAX- VOICE-PACKET-DELAY  USER) 

(SAFE-//  (USER-TOTAL-CONTROL-PACKET-DELAY  USER) 
(USER-NUMBER-OF-CONTROL- PACKETS  USER)) 
( USER-MAX-CONTROL-PACKET-DELAY  USER) 

(FIX  (USER-RATE  USER)) 

(FIX  (USER-PARTNER-RATE  USER))) 

(ALTER-USER  USER 

TOTAL-VOICE-PACKET-DELAY  0 
NUMBER-OF-VOICE- PACKETS  0 
MAX-VOICE-PACKET-DELAY  0 
TOTAL-CONTROL-PACKET-DELAY  0 
NUMBER-OF-CONTROL-PACKETS  0 
MAX-CONTROL-PACKET-DELAY  0))) 

( EVENT  # • USER-STATISTICS-COLLECTION 

(+  *TIME*  *USER-STATISTICS- INTERVAL*) 

USER)) 


Random  stuff. 


(DEFUN  RANDOM-INTER-TALK-SPURT-SILENCE  () 

0.0001) 

(DEFUN  PRINT-LINK-HISTOGRAMS  () 

(FORMAT  *LINK-STAT-STREAM*  ■ 

Av.  squared 

Link  Packets  Av.  wait  wait  Av.  flow  ") 

(LOOP  FOR  LINK  IN  *LINKS* 

DO 

(FORMAT  *LINK-STAT-STREAM* 

"  %  4D  8D  6,1,10$  6,1,10$  2,1,10$" 

(LINK-ID  LINK) 

( LINK-TOTAL-NUMBER-OF-PACKETS-SENT  LINK) 

(//  (LINK-TOTAL-PACKET-DELAY  LINK) 

(LINK-TOTAL-NUMBER-OF-PACKETS-SENT  LINK) ) 
(//  (LINK-TOTAL-SQUARED-PACKET-DELAY  LINK) 

(LINK-TOTAL-NUMBER-OF-PACKETS-SENT  LINK) ) 
(//  (LINK-TOTAL-NUMBER-OF-BITS-SENT  LINK) 
•TIME*)) 

(LOOP  FOR  I  FROM  0  TO  1 0 
DO 

(FORMAT 

•LINK-STAT-STREAM*  "  2,1,8$" 

(//  («  100.0 

(AREF  (LINK-PACKET-DELAY-HISTOGRAM  LINK) 
I)) 

(LINK-TOTAL-NUMBER-OF-PACKETS-SENT  LINK) ) ) ) 

)) 


V 

.v.v.v 
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